From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id QFd4IvC+nV4cJgAA0tVLHw (envelope-from ) for ; Mon, 20 Apr 2020 15:25:36 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id WNTMBva+nV6WFAAAbx9fmQ (envelope-from ) for ; Mon, 20 Apr 2020 15:25:42 +0000 Received: from arlo.cworth.org (arlo.cworth.org [50.126.95.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id A8087941D1C for ; Mon, 20 Apr 2020 15:25:40 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id E44476DE0F0C; Mon, 20 Apr 2020 08:25:36 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xyxW0yDaiohr; Mon, 20 Apr 2020 08:25:36 -0700 (PDT) Received: from arlo.cworth.org (localhost [IPv6:::1]) by arlo.cworth.org (Postfix) with ESMTP id AFE8E6DE0F21; Mon, 20 Apr 2020 08:25:35 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 5D92A6DE0F21 for ; Mon, 20 Apr 2020 08:25:33 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WwPEziggBv2f for ; Mon, 20 Apr 2020 08:25:30 -0700 (PDT) Received: from us-smtp-delivery-1.mimecast.com (us-smtp-1.mimecast.com [207.211.31.81]) by arlo.cworth.org (Postfix) with ESMTPS id 5A3DF6DE0F0C for ; Mon, 20 Apr 2020 08:25:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1587396329; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WW0g0LCfJABzT+tu76uqgtM5PrVG+FiHIYK89Z8i+eg=; b=i4FmyB3Oaw4o4431esDGo0aPH+o3e3JSHpdaYtlHb485dQUHhzUmcMXQ0eQ5YTtVrK9BW4 g6hlRATUPlwASDvh5p6hF7514HXt3yM5eEZDLU458FgGYjH6kDbIgucrqmXUm213D9qKoa bKrkgYCAvh7rLbyt4mwMvFx/+DZDGxw= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-163-mQLcZNzkOzu1xP3e6fTXXw-1; Mon, 20 Apr 2020 11:25:23 -0400 X-MC-Unique: mQLcZNzkOzu1xP3e6fTXXw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 006F91137848; Mon, 20 Apr 2020 15:25:22 +0000 (UTC) Received: from redhat.com (ovpn-116-138.rdu2.redhat.com [10.10.116.138]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 8618912656B; Mon, 20 Apr 2020 15:25:21 +0000 (UTC) Date: Mon, 20 Apr 2020 11:25:19 -0400 From: Don Zickus To: David Bremner Subject: Re: performance problems with notmuch new Message-ID: <20200420152519.gt73upo6mowhzsca@redhat.com> References: <20200415150801.h2mazyo37sspvech@redhat.com> <874ktku49b.fsf@tethera.net> <20200415173138.rn3ubtxo6mkracss@redhat.com> <87y2qwsdba.fsf@tethera.net> MIME-Version: 1.0 In-Reply-To: <87y2qwsdba.fsf@tethera.net> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Disposition: inline X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: notmuch@notmuchmail.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: notmuch-bounces@notmuchmail.org Sender: "notmuch" X-Scanner: scn0 X-Spam-Score: -0.11 Authentication-Results: aspmx1.migadu.com; dkim=fail (body hash did not verify) header.d=redhat.com header.s=mimecast20190719 header.b=i4FmyB3O; dmarc=fail reason="SPF not aligned (relaxed)" header.from=redhat.com (policy=none); spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 50.126.95.6 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Scan-Result: default: False [-0.11 / 13.00]; ARC_NA(0.00)[]; GENERIC_REPUTATION(0.00)[-0.4658589510029]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+a:c]; R_DKIM_REJECT(1.00)[redhat.com:s=mimecast20190719]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[notmuch@notmuchmail.org]; HAS_LIST_UNSUB(-0.01)[]; IP_REPUTATION_HAM(0.00)[asn: 27017(-0.19), country: US(-0.01), ip: 50.126.95.6(-0.47)]; URIBL_BLOCKED(0.00)[notmuchmail.org:email]; DKIM_TRACE(0.00)[redhat.com:-]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_MED(-0.20)[50.126.95.6:from]; MX_GOOD(-0.50)[cached: notmuchmail.org]; MAILLIST(-0.20)[mailman]; FORGED_SENDER_MAILLIST(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:27017, ipnet:50.126.64.0/18, country:US]; FROM_NEQ_ENVFROM(0.00)[dzickus@redhat.com,notmuch-bounces@notmuchmail.org]; RCVD_COUNT_SEVEN(0.00)[10]; DMARC_POLICY_SOFTFAIL(0.10)[redhat.com : SPF not aligned (relaxed),none] X-TUID: 655coIImxdmz On Wed, Apr 15, 2020 at 05:29:13PM -0300, David Bremner wrote: > Don Zickus writes: > > >> runs in about 30s here (i7 4770 / SSD). Replacing --small with --medium > >> takes about 10M (so a superlinear slowdown in wall clock time, since > >> that represents a 10x scale-up in the corpus size.). > > > > Hmm, for me --small was 35s and --medium was 32 minutes. This is on a > > i7-9750H / nvme. I would expect numbers similar to yours. > > I did another few tests test for --medium and they all take 7-9 minutes, > depending what else is going on on the machine. > Here's my breakdown of times (unfortunately a bit of hand editing is > needed to clean up the warnings) Sorry for the delay. I re-ran the test today and see similar numbers to yours now. Roughly 6m30s per run (3 runs). T00-new.sh: Testing notmuch new [0.4 medium] Wall(s) Usr(s) Sys(s) Res(K) In/Out(512B) Initial notmuch new 61.86 59.13 2.53 228124 2024/1087672 notmuch new #2 0.01 0.00 0.00 9136 0/160 notmuch new #3 0.00 0.00 0.00 9068 0/8 notmuch new #4 0.00 0.00 0.00 8940 0/8 notmuch new #5 0.00 0.00 0.00 9060 0/8 notmuch new #6 0.00 0.00 0.00 8932 0/8 new (7500 mv) 44.76 32.67 12.02 186156 0/541024 new (7500 mv back) 47.82 34.81 12.91 187240 0/500000 new (7500 cp) 26.41 20.06 6.28 183448 0/378192 T01-dump-restore.sh: Testing dump and restore [0.4 medium] Wall(s) Usr(s) Sys(s) Res(K) In/Out(512B) 3.35 1.82 1.31 11992 624/31976 dump * 0.60 0.57 0.02 11356 0/4344 restore * 0.69 0.62 0.06 9064 0/0 T02-tag.sh: Testing tagging [0.4 medium] Wall(s) Usr(s) Sys(s) Res(K) In/Out(512B) tag * +new_tag 39.96 24.99 14.74 80048 16/282944 tag * +existing_tag 0.00 0.00 0.00 8968 0/0 tag * -existing_tag 29.98 18.18 11.70 20040 0/208264 tag * -missing_tag 0.00 0.00 0.00 9036 0/0 T03-reindex.sh: Testing reindexing [0.4 medium] Wall(s) Usr(s) Sys(s) Res(K) In/Out(512B) reindex * 40.56 33.46 7.02 228908 0/459240 reindex * 39.91 33.10 6.75 223768 112/378824 reindex * 40.04 33.10 6.89 223724 0/334824 T04-thread-subquery.sh: Testing thread subqueries [0.4 medium] Wall(s) Usr(s) Sys(s) Res(K) In/Out(512B) search thread:{} ... 0.29 0.26 0.02 23256 0/24 search thread:{} ... 0.27 0.25 0.02 23296 0/24 search thread:{} ... 0.29 0.27 0.02 23192 0/24 The only thing I can think of is my fstrim service runs 1x / week on Monday at midnight and maybe that helped clean things up?? Perhaps I should increase that frequency or run it manually when things go bad. Thanks for the feedback! Cheers, Don