From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id aCV4E6Vul164EgAA0tVLHw (envelope-from ) for ; Wed, 15 Apr 2020 20:29:25 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id uOROKKhul15GEgAA1q6Kng (envelope-from ) for ; Wed, 15 Apr 2020 20:29:28 +0000 Received: from arlo.cworth.org (arlo.cworth.org [50.126.95.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 55951941A95 for ; Wed, 15 Apr 2020 20:29:25 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 7445B6DE092F; Wed, 15 Apr 2020 13:29:20 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Pr4ioVUMteB1; Wed, 15 Apr 2020 13:29:19 -0700 (PDT) Received: from arlo.cworth.org (localhost [IPv6:::1]) by arlo.cworth.org (Postfix) with ESMTP id 640F06DE0314; Wed, 15 Apr 2020 13:29:19 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 070116DE0314 for ; Wed, 15 Apr 2020 13:29:18 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VgNXNj_FAhvl for ; Wed, 15 Apr 2020 13:29:16 -0700 (PDT) Received: from fethera.tethera.net (fethera.tethera.net [198.245.60.197]) by arlo.cworth.org (Postfix) with ESMTPS id DA3B96DE023B for ; Wed, 15 Apr 2020 13:29:16 -0700 (PDT) Received: from remotemail by fethera.tethera.net with local (Exim 4.92) (envelope-from ) id 1jOoep-0007mK-ET; Wed, 15 Apr 2020 16:29:15 -0400 Received: (nullmailer pid 270122 invoked by uid 1000); Wed, 15 Apr 2020 20:29:13 -0000 From: David Bremner To: Don Zickus Subject: Re: performance problems with notmuch new In-Reply-To: <20200415173138.rn3ubtxo6mkracss@redhat.com> References: <20200415150801.h2mazyo37sspvech@redhat.com> <874ktku49b.fsf@tethera.net> <20200415173138.rn3ubtxo6mkracss@redhat.com> Date: Wed, 15 Apr 2020 17:29:13 -0300 Message-ID: <87y2qwsdba.fsf@tethera.net> MIME-Version: 1.0 X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: notmuch@notmuchmail.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: notmuch-bounces@notmuchmail.org Sender: "notmuch" X-Scanner: scn0 X-Spam-Score: -1.01 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 50.126.95.6 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Scan-Result: default: False [-1.01 / 13.00]; RCVD_TLS_LAST(0.00)[]; GENERIC_REPUTATION(0.00)[-0.45195320213838]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; IP_REPUTATION_HAM(0.00)[asn: 27017(-0.18), country: US(-0.01), ip: 50.126.95.6(-0.45)]; R_SPF_ALLOW(-0.20)[+a]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[notmuch@notmuchmail.org]; ARC_NA(0.00)[]; HAS_LIST_UNSUB(-0.01)[]; DMARC_NA(0.00)[tethera.net]; SPF_REPUTATION_HAM(0.00)[-0.45019802324045]; MX_GOOD(-0.50)[cached: notmuchmail.org]; RCPT_COUNT_TWO(0.00)[2]; MAILLIST(-0.20)[mailman]; FORGED_SENDER_MAILLIST(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:27017, ipnet:50.126.64.0/18, country:US]; FROM_NEQ_ENVFROM(0.00)[david@tethera.net,notmuch-bounces@notmuchmail.org]; RCVD_COUNT_SEVEN(0.00)[8] X-TUID: NZ1OBMP4lcOZ Don Zickus writes: >> runs in about 30s here (i7 4770 / SSD). Replacing --small with --medium >> takes about 10M (so a superlinear slowdown in wall clock time, since >> that represents a 10x scale-up in the corpus size.). > > Hmm, for me --small was 35s and --medium was 32 minutes. This is on a > i7-9750H / nvme. I would expect numbers similar to yours. I did another few tests test for --medium and they all take 7-9 minutes, depending what else is going on on the machine. Here's my breakdown of times (unfortunately a bit of hand editing is needed to clean up the warnings) performance-test/notmuch-time-test --medium T00-new.sh: Testing notmuch new [0.4 medium] Wall(s) Usr(s) Sys(s) Res(K) In/Out(512B) Initial notmuch new 66.29 62.22 2.82 241148 0/1089784 notmuch new #2 0.03 0.00 0.00 9864 0/160 notmuch new #3 0.00 0.00 0.00 9292 0/8 notmuch new #4 0.00 0.00 0.00 9556 0/8 notmuch new #5 0.00 0.00 0.00 9396 0/8 notmuch new #6 0.00 0.00 0.00 9316 0/8 new (7500 mv) 49.02 35.88 12.59 185152 0/392720 new (7500 mv back) 59.41 43.04 15.88 185888 0/413824 new (7500 cp) 36.09 26.55 9.02 182160 0/411840 T01-dump-restore.sh: Testing dump and restore [0.4 medium] Wall(s) Usr(s) Sys(s) Res(K) In/Out(512B) load nmbug tags 5.37 2.09 1.63 12172 0/31864 dump * 0.63 0.58 0.04 11756 0/4344 restore * 0.72 0.65 0.06 9572 0/0 T02-tag.sh: Testing tagging [0.4 medium] Wall(s) Usr(s) Sys(s) Res(K) In/Out(512B) tag * +new_tag 54.45 31.93 20.26 86380 8/250512 tag * +existing_tag 0.00 0.00 0.00 9396 0/0 tag * -existing_tag 46.73 26.07 20.08 20580 0/284248 tag * -missing_tag 0.00 0.00 0.00 9316 0/0 T03-reindex.sh: Testing reindexing [0.4 medium] Wall(s) Usr(s) Sys(s) Res(K) In/Out(512B) reindex * 78.39 63.06 14.31 229204 0/546136 reindex * 70.52 56.68 13.12 223980 0/333608 reindex * 78.02 62.61 14.92 225160 0/374424 T04-thread-subquery.sh: Testing thread subqueries [0.4 medium] Wall(s) Usr(s) Sys(s) Res(K) In/Out(512B) search thread:{} ... 0.37 0.33 0.04 26508 0/24 search thread:{} ... 0.38 0.33 0.04 23592 0/24 search thread:{} ... 0.37 0.34 0.03 26612 0/24 415.28user 128.24system 9:13.07elapsed 98%CPU (0avgtext+0avgdata 241148maxresident)k 8inputs+7294896outputs (0major+458590minor)pagefaults 0swaps