From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id QC/3Bwe6nV5XOgAA0tVLHw (envelope-from ) for ; Mon, 20 Apr 2020 15:04:39 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id YDNLAwy6nV7bNwAAB5/wlQ (envelope-from ) for ; Mon, 20 Apr 2020 15:04:44 +0000 Received: from arlo.cworth.org (arlo.cworth.org [50.126.95.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id ABF3E941D2C for ; Mon, 20 Apr 2020 15:04:42 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 023976DE0F50; Mon, 20 Apr 2020 08:04:39 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fSHjU90E4uaJ; Mon, 20 Apr 2020 08:04:38 -0700 (PDT) Received: from arlo.cworth.org (localhost [IPv6:::1]) by arlo.cworth.org (Postfix) with ESMTP id D82676DE0E7F; Mon, 20 Apr 2020 08:04:37 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 5CDCE6DE0E7F for ; Mon, 20 Apr 2020 08:04:36 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ToK1Ycbl2dES for ; Mon, 20 Apr 2020 08:04:34 -0700 (PDT) Received: from mx4.nic.fr (mx4.nic.fr [192.134.4.12]) by arlo.cworth.org (Postfix) with ESMTPS id DF06D6DE0C67 for ; Mon, 20 Apr 2020 08:04:33 -0700 (PDT) Received: from mx4.nic.fr (localhost [127.0.0.1]) by mx4.nic.fr (Postfix) with SMTP id F39CF28018D; Mon, 20 Apr 2020 17:04:30 +0200 (CEST) Received: by mx4.nic.fr (Postfix, from userid 500) id ED0E62805E4; Mon, 20 Apr 2020 17:04:30 +0200 (CEST) Received: from relay01.prive.nic.fr (relay01.prive.nic.fr [IPv6:2001:67c:2218:15::11]) by mx4.nic.fr (Postfix) with ESMTP id E50C528018D; Mon, 20 Apr 2020 17:04:30 +0200 (CEST) Received: from kakou.tech.prive.nic.fr (kakou.users.prive.nic.fr [10.10.86.119]) by relay01.prive.nic.fr (Postfix) with ESMTP id E1304642C582; Mon, 20 Apr 2020 17:04:30 +0200 (CEST) Received: by kakou.tech.prive.nic.fr (Postfix, from userid 1000) id D49121515B4B; Mon, 20 Apr 2020 15:04:30 +0000 (UTC) From: Kim Minh Kaplan To: notmuch@notmuchmail.org Subject: Re: performance problems with notmuch new In-Reply-To: <1587211167-ner-6.432@LappyL520> References: <20200415150801.h2mazyo37sspvech@redhat.com> <1587211167-ner-6.432@LappyL520> Date: Mon, 20 Apr 2020 15:04:30 +0000 Message-ID: <875zdukxkx.fsf@kakou.tech.prive.nic.fr> MIME-Version: 1.0 X-Bogosity: No, tests=bogofilter, spamicity=0.297217, version=1.2.2 X-PMX-Version: 6.0.0.2142326, Antispam-Engine: 2.7.2.2107409, Antispam-Data: 2019.11.5.63017 X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Don Zickus Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: notmuch-bounces@notmuchmail.org Sender: "notmuch" X-Scanner: scn0 X-Spam-Score: -1.21 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 50.126.95.6 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Scan-Result: default: False [-1.21 / 13.00]; ARC_NA(0.00)[]; GENERIC_REPUTATION(0.00)[-0.46577403303489]; RCVD_COUNT_SEVEN(0.00)[11]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+a:c]; IP_REPUTATION_HAM(0.00)[asn: 27017(-0.19), country: US(-0.01), ip: 50.126.95.6(-0.47)]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[notmuch@notmuchmail.org]; DMARC_NA(0.00)[afnic.fr]; HAS_LIST_UNSUB(-0.01)[]; FORGED_SENDER_MAILLIST(0.00)[]; URIBL_BLOCKED(0.00)[notmuchmail.org:email]; RCVD_TLS_LAST(0.00)[]; MX_GOOD(-0.50)[cached: notmuchmail.org]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_MED(-0.20)[50.126.95.6:from]; MAILLIST(-0.20)[mailman]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:27017, ipnet:50.126.64.0/18, country:US]; TAGGED_FROM(0.00)[nomuch]; FROM_NEQ_ENVFROM(0.00)[kimminh.kaplan@afnic.fr,notmuch-bounces@notmuchmail.org] X-TUID: z24R00k5fS5o Franz Fellner writes: > I also suffer from bad performance of notmuch new. > I used notmuch some years ago and notmuch new always felt instantanious. > Had to stop using it because internet was too slow to sync my mails :/ > Now (with better internet and a completely new setup using mbsync) indexing one mail takes at least 10 seconds, sometimes even more. > It can go into minutes when I get lots of mail (~30...). > When I run it after a reboot I can have breakfast while notmuch starts up... > This is all on spinning rust. I thought of getting an SSD but not in the near future. > > What I observe during that time: notmuch doesn't really need much CPU. > iotop shows constant read and write with extremely low rates, under 1MB/sec. > So I think it might be an issue in xapian? Hello, I had a similar performance issue working with a very large email set (in the million count). Amongst those, I have crons that run about every minute and email back a report, mostly identical every time with some thousands words. Apparently indexing many times mostly identical emails triggers a pathological behaviour. Moving these email out of my mail directories brought back acceptable performances. We are speaking in the 10,000 to 100,000 files here. Kim Minh. > If there is anything I can do to help debug this please tell me > > Franz > > P.S.: > @David: Sorry for writing only to you. > GMail web interface only added you as recipient and not the list...