From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id KOSVCKJyo167JgAA0tVLHw (envelope-from ) for ; Fri, 24 Apr 2020 23:13:38 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id 6O3vJahyo161TAAAB5/wlQ (envelope-from ) for ; Fri, 24 Apr 2020 23:13:44 +0000 Received: from arlo.cworth.org (arlo.cworth.org [50.126.95.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 1ABFD940D4F for ; Fri, 24 Apr 2020 23:13:43 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 8883F6DE13B8; Fri, 24 Apr 2020 16:13:39 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rBdUi_Q7O_w7; Fri, 24 Apr 2020 16:13:38 -0700 (PDT) Received: from arlo.cworth.org (localhost [IPv6:::1]) by arlo.cworth.org (Postfix) with ESMTP id 1BF9E6DE13DA; Fri, 24 Apr 2020 16:13:38 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 331A36DE13DA for ; Fri, 24 Apr 2020 16:13:37 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id N4pBFoppVsSC for ; Fri, 24 Apr 2020 16:13:35 -0700 (PDT) Received: from fethera.tethera.net (fethera.tethera.net [198.245.60.197]) by arlo.cworth.org (Postfix) with ESMTPS id 3EB986DE13B8 for ; Fri, 24 Apr 2020 16:13:35 -0700 (PDT) Received: from remotemail by fethera.tethera.net with local (Exim 4.92) (envelope-from ) id 1jS7Vk-0004JZ-HO; Fri, 24 Apr 2020 19:13:32 -0400 Received: (nullmailer pid 3072691 invoked by uid 1000); Fri, 24 Apr 2020 23:13:30 -0000 From: David Bremner To: Franz Fellner , Olly Betts Subject: Re: performance problems with notmuch new In-Reply-To: <1587753355-ner-5.891@LappyL520> References: <20200415150801.h2mazyo37sspvech@redhat.com> <1587211167-ner-6.432@LappyL520> <87imhup6kr.fsf@tethera.net> <20200422232130.GH28897@survex.com> <1587753355-ner-5.891@LappyL520> Date: Fri, 24 Apr 2020 20:13:30 -0300 Message-ID: <87k124jx45.fsf@tethera.net> MIME-Version: 1.0 X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: notmuch@notmuchmail.org, Xapian Discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: notmuch-bounces@notmuchmail.org Sender: "notmuch" X-Scanner: scn0 X-Spam-Score: 0.49 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 50.126.95.6 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Scan-Result: default: False [0.49 / 13.00]; FORGED_SENDER_MAILLIST(0.00)[]; GENERIC_REPUTATION(0.00)[-0.46308381620021]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+a:c]; IP_REPUTATION_HAM(0.00)[asn: 27017(-0.19), country: US(-0.00), ip: 50.126.95.6(-0.46)]; DWL_DNSWL_FAIL(0.00)[50.126.95.6:server fail]; MX_GOOD(-0.50)[cached: notmuchmail.org]; MAILLIST(-0.20)[mailman]; FREEMAIL_TO(0.00)[gmail.com,survex.com]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:27017, ipnet:50.126.64.0/18, country:US]; MID_RHS_MATCH_FROM(0.00)[]; FROM_NEQ_ENVFROM(0.00)[david@tethera.net,notmuch-bounces@notmuchmail.org]; ARC_NA(0.00)[]; URIBL_BLOCKED(0.00)[notmuchmail.org:email,survex.com:email]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[notmuch@notmuchmail.org]; HAS_LIST_UNSUB(-0.01)[]; DMARC_NA(0.00)[tethera.net]; DNSWL_BLOCKED(0.00)[50.126.95.6:from]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_SEVEN(0.00)[8]; SUSPICIOUS_RECIPS(1.50)[] X-TUID: uIYwiCoBeHOm Franz Fellner writes: > > On Thu Apr 23 00:21:30 2020, Olly Betts wrote: >> Then I'd try compacting the database (I think there's a "notmuch >> compact" subcommand to do this). > And there we go. Cured the issues. Dropped the very first indexing > from several minutes to 1.5 seconds on the desktop. ?!?! This is a > really new setup and I suffered from bad performance from the very > first notmuch new after the initial indexing. Is it really needed to > run notmch compact directly after the initial notmuch new? Desktop > currently has 38502 messages indexed, in case that matters. That's a fairly small number of messages by the usual notmuch standards. It's possible there is something pathological about the interaction of notmuch and your particular set of mail. Running the performance test suite on your machine would be one way of telling. Also, the size of your notmuch database on disk; really huge databases are a sign that something is going wrong. I think most people find the database is 1/3 to 1/2 the size of the mail it is indexing, but we have had some odd cases where the database is 2 or 3 times the mail, and that was an indexing bug. d