From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id OHBvJHuznV5GYAAA0tVLHw (envelope-from ) for ; Mon, 20 Apr 2020 14:36:43 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id MCFOH4CznV49LgAAB5/wlQ (envelope-from ) for ; Mon, 20 Apr 2020 14:36:48 +0000 Received: from arlo.cworth.org (arlo.cworth.org [50.126.95.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id CD303941B5E for ; Mon, 20 Apr 2020 14:36:46 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 5478B6DE0F39; Mon, 20 Apr 2020 07:36:43 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ov3QjbM3YCBX; Mon, 20 Apr 2020 07:36:42 -0700 (PDT) Received: from arlo.cworth.org (localhost [IPv6:::1]) by arlo.cworth.org (Postfix) with ESMTP id 408FE6DE0B64; Mon, 20 Apr 2020 07:36:42 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id A09AE6DE0B64 for ; Mon, 20 Apr 2020 07:36:41 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qEncUI4Vhifi for ; Mon, 20 Apr 2020 07:36:41 -0700 (PDT) Received: from fethera.tethera.net (fethera.tethera.net [198.245.60.197]) by arlo.cworth.org (Postfix) with ESMTPS id EEE136DE0B36 for ; Mon, 20 Apr 2020 07:36:40 -0700 (PDT) Received: from remotemail by fethera.tethera.net with local (Exim 4.92) (envelope-from ) id 1jQXXJ-0002vM-J9; Mon, 20 Apr 2020 10:36:37 -0400 Received: (nullmailer pid 1671437 invoked by uid 1000); Mon, 20 Apr 2020 14:36:36 -0000 From: David Bremner To: Franz Fellner , Don Zickus , notmuch@notmuchmail.org Subject: Re: performance problems with notmuch new In-Reply-To: <1587211167-ner-6.432@LappyL520> References: <20200415150801.h2mazyo37sspvech@redhat.com> <1587211167-ner-6.432@LappyL520> X-List-To: notmuch Date: Mon, 20 Apr 2020 11:36:36 -0300 Message-ID: <87imhup6kr.fsf@tethera.net> MIME-Version: 1.0 X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: xapian-discuss@lists.xapian.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: notmuch-bounces@notmuchmail.org Sender: "notmuch" X-Scanner: scn0 X-Spam-Score: 0.29 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 50.126.95.6 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Scan-Result: default: False [0.29 / 13.00]; FORGED_SENDER_MAILLIST(0.00)[]; GENERIC_REPUTATION(0.00)[-0.46577255217734]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+a:c]; IP_REPUTATION_HAM(0.00)[asn: 27017(-0.19), country: US(-0.01), ip: 50.126.95.6(-0.47)]; MX_GOOD(-0.50)[cached: notmuchmail.org]; RCVD_IN_DNSWL_MED(-0.20)[50.126.95.6:from]; MAILLIST(-0.20)[mailman]; FREEMAIL_TO(0.00)[gmail.com,redhat.com,notmuchmail.org]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:27017, ipnet:50.126.64.0/18, country:US]; MID_RHS_MATCH_FROM(0.00)[]; FROM_NEQ_ENVFROM(0.00)[david@tethera.net,notmuch-bounces@notmuchmail.org]; ARC_NA(0.00)[]; URIBL_BLOCKED(0.00)[notmuchmail.org:email]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[notmuch@notmuchmail.org]; HAS_LIST_UNSUB(-0.01)[]; DMARC_NA(0.00)[tethera.net]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_SEVEN(0.00)[8]; SUSPICIOUS_RECIPS(1.50)[] X-TUID: 6iRY4JmGMWNA Franz Fellner writes: > I also suffer from bad performance of notmuch new. I used notmuch > some years ago and notmuch new always felt instantanious. Had to stop > using it because internet was too slow to sync my mails :/ Now (with > better internet and a completely new setup using mbsync) indexing one > mail takes at least 10 seconds, sometimes even more. It can go into > minutes when I get lots of mail (~30...). When I run it after a > reboot I can have breakfast while notmuch starts up... This is all on > spinning rust. I thought of getting an SSD but not in the near future. I do have at least one spinning rust configuration with about 300k messages, and notmuch new is still fast there. > What I observe during that time: notmuch doesn't really need much CPU. > iotop shows constant read and write with extremely low rates, under > 1MB/sec. So I think it might be an issue in xapian? > Just in case one of the xapian experts can suggest some kind of test for why you might be seeing this behaviour, I've included the xapian list in CC. > If there is anything I can do to help debug this please tell me What kind of filesystem do you have on your spinning rust? d