From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id 2kxmHIY4o17PdQAA0tVLHw (envelope-from ) for ; Fri, 24 Apr 2020 19:05:42 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id aOESA404o15KLQAA1q6Kng (envelope-from ) for ; Fri, 24 Apr 2020 19:05:49 +0000 Received: from arlo.cworth.org (arlo.cworth.org [50.126.95.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id C0175943140 for ; Fri, 24 Apr 2020 19:05:47 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 925356DE13CC; Fri, 24 Apr 2020 12:05:44 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id L3ZcUXLvZMFP; Fri, 24 Apr 2020 12:05:44 -0700 (PDT) Received: from arlo.cworth.org (localhost [IPv6:::1]) by arlo.cworth.org (Postfix) with ESMTP id 6F3FF6DE13BA; Fri, 24 Apr 2020 12:05:43 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id E3C016DE13BA for ; Fri, 24 Apr 2020 12:05:41 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pwGOlNTkjyp5 for ; Fri, 24 Apr 2020 12:05:40 -0700 (PDT) Received: from us-smtp-delivery-1.mimecast.com (us-smtp-1.mimecast.com [207.211.31.81]) by arlo.cworth.org (Postfix) with ESMTPS id 0B81D6DE13B8 for ; Fri, 24 Apr 2020 12:05:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1587755138; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kMCBejh3v5ovDEEPWwm25igji/JFLYfaXuXUBh4+k24=; b=UQjUQWXP5Rh1pzMP0Oa8H8XZrn2nKBsfZB7Oi/FYxqRxqftYe0E92DYZakMBR6KgaRYRUU UKiVyWRFNh3/HKt9YVJXntt6F49TuJ2/mwLeE3oIbWUNNhom6Gc9JxgeEOydosYG2ZdIZI NCi+jfcv6+CL4pAhLIdJPJ5EZVLOny4= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-115-nom4svqAMaqb3YJ7zReLeQ-1; Fri, 24 Apr 2020 15:05:32 -0400 X-MC-Unique: nom4svqAMaqb3YJ7zReLeQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A57F6835B43; Fri, 24 Apr 2020 19:05:30 +0000 (UTC) Received: from redhat.com (ovpn-116-138.rdu2.redhat.com [10.10.116.138]) by smtp.corp.redhat.com (Postfix) with ESMTPS id B3F0B60CD3; Fri, 24 Apr 2020 19:05:28 +0000 (UTC) Date: Fri, 24 Apr 2020 15:05:26 -0400 From: Don Zickus To: David Bremner Subject: Re: performance problems with notmuch new Message-ID: <20200424190526.gfypvd4aooe34y6t@redhat.com> References: <20200415150801.h2mazyo37sspvech@redhat.com> <874ktku49b.fsf@tethera.net> <20200415173138.rn3ubtxo6mkracss@redhat.com> <87y2qwsdba.fsf@tethera.net> <20200420152519.gt73upo6mowhzsca@redhat.com> <87y2qljhlk.fsf@tethera.net> MIME-Version: 1.0 In-Reply-To: <87y2qljhlk.fsf@tethera.net> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Disposition: inline X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: notmuch@notmuchmail.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: notmuch-bounces@notmuchmail.org Sender: "notmuch" X-Scanner: scn0 X-Spam-Score: 0.09 Authentication-Results: aspmx1.migadu.com; dkim=fail (body hash did not verify) header.d=redhat.com header.s=mimecast20190719 header.b=UQjUQWXP; dmarc=fail reason="SPF not aligned (relaxed)" header.from=redhat.com (policy=none); spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 50.126.95.6 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Scan-Result: default: False [0.09 / 13.00]; GENERIC_REPUTATION(0.00)[-0.46239675927959]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+a:c]; IP_REPUTATION_HAM(0.00)[asn: 27017(-0.18), country: US(-0.00), ip: 50.126.95.6(-0.46)]; DWL_DNSWL_BLOCKED(0.00)[50.126.95.6:from]; R_DKIM_REJECT(1.00)[redhat.com:s=mimecast20190719]; MX_GOOD(-0.50)[cached: notmuchmail.org]; RCPT_COUNT_TWO(0.00)[2]; DKIM_TRACE(0.00)[redhat.com:-]; MAILLIST(-0.20)[mailman]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:27017, ipnet:50.126.64.0/18, country:US]; MID_RHS_MATCH_FROM(0.00)[]; FROM_NEQ_ENVFROM(0.00)[dzickus@redhat.com,notmuch-bounces@notmuchmail.org]; ARC_NA(0.00)[]; URIBL_BLOCKED(0.00)[notmuchmail.org:email]; FROM_HAS_DN(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[notmuch@notmuchmail.org]; HAS_LIST_UNSUB(-0.01)[]; DNSWL_BLOCKED(0.00)[50.126.95.6:from]; RCVD_COUNT_SEVEN(0.00)[10]; FORGED_SENDER_MAILLIST(0.00)[]; DMARC_POLICY_SOFTFAIL(0.10)[redhat.com : SPF not aligned (relaxed),none] X-TUID: uMsZLowXhPI5 On Fri, Apr 24, 2020 at 07:36:23AM -0300, David Bremner wrote: > Don Zickus writes: > > > > > The only thing I can think of is my fstrim service runs 1x / week on Monday > > at midnight and maybe that helped clean things up?? Perhaps I should > > increase that frequency or run it manually when things go bad. > > > > Is notmuch new still slow on your mail store, or did that mysteriously > improve as well? It improved slightly. Perhaps my expectations are too high. After about a 1000, the speed slows down significantly. Outside of the performance testing, after about 1000 emails, the processing slows down to about 10/second. Of course I am using this in a strange context. I am deleting emails locally and then running 'notmuch new' to clean up the database. notmuch search --output=files |xargs -l rm notmuch new and sometimes I delete a couple thousand emails from my mailing list folder in neomutt and while exiting the folder it takes a long while to sync up with notmuch. So maybe my use case is unusual and the slowness is expected. Cheers, Don