From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2.migadu.com ([2001:41d0:403:4876::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms1.migadu.com with LMTPS id sJE0G1onU2Ye+gAAe85BDQ:P1 (envelope-from ) for ; Sun, 26 May 2024 14:13:14 +0200 Received: from aspmx1.migadu.com ([2001:41d0:403:4876::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2.migadu.com with LMTPS id sJE0G1onU2Ye+gAAe85BDQ (envelope-from ) for ; Sun, 26 May 2024 14:13:14 +0200 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 135.181.149.255 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1716725594; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-owner:list-unsubscribe:list-subscribe:list-post; bh=FESO4CTyaSAmKUXVM8RDwa7NcqG6C20BDIX9DAr/VLA=; b=rwQ6oYYQuisUMzjIu1pWJiRAlc10pLjDZr3sHT6JxaG+lvQq2HE66SbWSw3Mc3beelPtkw KI7WoeEUO39z3WMhILPWi2SpEzGx2E+B2+L25smrqKQ0A+RxfHRFS5mA8zZLFt530n6ezq xvvHLWUTSAuhuZy1xH4CgWBVaTWP9GtuR8ubUv6pivF5KH/lWCO6Zczo2BknOu3n84gSJ1 nf4e1/CAe/Z2lc5mCf09xP2nBn6B+5IclEuqsZWXwuEu4zkdvtr8+FjWXsdmOXS1n+kS53 Gu6FBTLmb+9vlUvz1CzxZrK24ggsaP7ZrpxucVo7BpVVsuabPTnIeNoAmcju4g== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 135.181.149.255 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org; dmarc=none ARC-Seal: i=1; s=key1; d=yhetil.org; t=1716725594; a=rsa-sha256; cv=none; b=iXfoCJ0Dase8h5Zfwh3aAGFDWoZG2wIhmhwzcN3XFoKPdukYzvrmxXuHDtmFUSYnNxUMIt 08iR1FIF3lpn24xbc6qmOW2kfpNx4n3QwbtC5oWj+9JjqGzwWnblYhmX1vglVJTBBilNmk 60mOrjzvCBHJgMqHb6bH9IsFuJPn7DA1oTQxQ2UZYseXADcCQNLBFLK2Oqi17KweBD0oD4 XOnED0i8U9KJwkUlqNF9kT5RomWqbTROMhY4iAvB5NyoBj/ukBIypq0fc0WUBQZqLEW88M 2tNRnAoluNqpGLAwDoZ9BmFN5SgavUdQfjwueSjbEJqNWsCNDND2oaqqEX96Jw== Received: from mail.notmuchmail.org (yantan.tethera.net [135.181.149.255]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 266937A234 for ; Sun, 26 May 2024 14:13:13 +0200 (CEST) Received: from yantan.tethera.net (localhost [127.0.0.1]) by mail.notmuchmail.org (Postfix) with ESMTP id 246E15E50C; Sun, 26 May 2024 12:13:04 +0000 (UTC) Received: from phubs.tethera.net (phubs.tethera.net [192.99.9.157]) by mail.notmuchmail.org (Postfix) with ESMTPS id 8A2805E502 for ; Sun, 26 May 2024 12:13:01 +0000 (UTC) Received: from tethera.net (fctnnbsc51w-159-2-211-58.dhcp-dynamic.fibreop.nb.bellaliant.net [159.2.211.58]) by phubs.tethera.net (Postfix) with ESMTPS id 9FC4B180066; Sun, 26 May 2024 09:12:58 -0300 (ADT) Received: (nullmailer pid 646518 invoked by uid 1000); Sun, 26 May 2024 12:12:57 -0000 From: David Bremner To: Matthew Schauer , notmuch@notmuchmail.org Subject: Re: Xapian commits unexpectedly slow In-Reply-To: <6a695b68-6f21-2125-d087-55816e07fffa@e10x.net> References: <4b3b642b-8f5b-4e8c-9f29-76d393d45fd6@e10x.net> <87tv5kvv66.fsf@tethera.net> <6a695b68-6f21-2125-d087-55816e07fffa@e10x.net> Date: Sun, 26 May 2024 09:12:57 -0300 Message-ID: <8734q422ly.fsf@tethera.net> MIME-Version: 1.0 Message-ID-Hash: 62AUJKF4PXDZASJNYLHPF3BCDTY2EWDA X-Message-ID-Hash: 62AUJKF4PXDZASJNYLHPF3BCDTY2EWDA X-MailFrom: david@tethera.net X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-notmuch.notmuchmail.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.3 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Migadu-Country: DE X-Migadu-Flow: FLOW_IN X-Spam-Score: -1.46 X-Migadu-Queue-Id: 266937A234 X-Migadu-Scanner: mx10.migadu.com X-Migadu-Spam-Score: -1.46 X-TUID: Nyb17+bxEWmA Matthew Schauer writes: > > Nifty! Here are the results -- I assume you know how to interpret them > better than I do: > > T00-new.sh: Testing notmuch new [0.4 large] > Wall(s) Usr(s) Sys(s) Res(K) In/Out(512B) > Initial notmuch new 1163.05 854.26 45.97 444304 2343120/13645200 > notmuch new #2 2.23 0.02 0.03 9384 2144/8 > notmuch new #3 0.01 0.01 0.00 9460 0/8 > notmuch new #4 0.01 0.01 0.00 9428 0/8 > notmuch new #5 0.01 0.00 0.00 9468 0/8 > notmuch new #6 0.01 0.01 0.00 9692 0/8 > new (52374 mv) 1351.01 537.75 235.45 959524 1027288/8531616 > new (52374 mv back) 834.15 489.27 213.97 967040 184/4754016 > new (52374 cp) 747.23 284.03 105.51 941992 0/4007120 > Apologies, it looks like I never replied to this thread. Probably you are not longer interested, but I can make a few observations, mainly that there are a few relevant improvements in later notmuch. 1) This is about 3x slower than my current benchmark machine [1]. My current machine is probably 4 years newer, so I would expect some improvement in performance. 2) I don't know if this is typical for spinning rust, but about about 25% of the time is (apparently) IO wait, since it it does not show up in CPU time. I do have access to a machine with both SSD and spinning rust, but the latter is in some complicated RAID formation, so I don't know how representative the results would be. 3) Some time after you reported these issues I implemented an "autocommit" parameter, which should should help avoid large Xapian large commits. 4) Your results show that notmuch new could be extra slow when dealing with moving files on disk. This should be somewhat improved by changes in notmuch 0.32 (I also see fairly dramatic impovements in notmuch-reindex relative to notmuch new, but the underlying cause is less clear). [1] e.g. https://notmuchmail.org/perf-test-results/2024-05-26-minkowski/