From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id QMzOBo8xo15XVAAA0tVLHw (envelope-from ) for ; Fri, 24 Apr 2020 18:35:59 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id 4KwlIJUxo16KewAAB5/wlQ (envelope-from ) for ; Fri, 24 Apr 2020 18:36:05 +0000 Received: from arlo.cworth.org (arlo.cworth.org [50.126.95.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 2DA3A9430E3 for ; Fri, 24 Apr 2020 18:36:04 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id E23D16DE13B8; Fri, 24 Apr 2020 11:35:59 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OFZbEEd35iIw; Fri, 24 Apr 2020 11:35:59 -0700 (PDT) Received: from arlo.cworth.org (localhost [IPv6:::1]) by arlo.cworth.org (Postfix) with ESMTP id 9CA196DE13BA; Fri, 24 Apr 2020 11:35:58 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id B9FDE6DE13BA for ; Fri, 24 Apr 2020 11:35:56 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1juQP60aN9Rr for ; Fri, 24 Apr 2020 11:35:55 -0700 (PDT) Received: from mail-wr1-f43.google.com (mail-wr1-f43.google.com [209.85.221.43]) by arlo.cworth.org (Postfix) with ESMTPS id 7D7AF6DE13B8 for ; Fri, 24 Apr 2020 11:35:55 -0700 (PDT) Received: by mail-wr1-f43.google.com with SMTP id t14so12106478wrw.12 for ; Fri, 24 Apr 2020 11:35:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:subject:to:cc:references:in-reply-to:mime-version:date :message-id; bh=oYcZFNj7C4gqFVxy8ocUWkgd0xBG/Pe8ToC0wSpkruM=; b=krEqc3wEDlIlWWfbdZunl7Ffv20Ag01F67FxpnYr42XeXpfJ/R1Ud9/C/zdFkK8KYQ hj0/EzCxVMMostwUwRg5PdJILMtaTqI7C/N2pwJX7QiVDDol1Xt2f9gwnnqActqeTWb5 5I4+kgZtdbak0thcRjesaYqyhPit3Z5IOPQNxiKhp6xOLie2xKs9+9nlYoWhMkBtwK/3 rz01SLLOj8dZ6NH6+ck82mUeq6HCS5h9LH2qheOjCjeYSzqHerJVqvaXQLjTueRBM5lS qUCkXsHMie59oDPQ/jdzo7Zzte2yYIyeEgDotnQP4XGnHi22w5Qph9555bkXkSPXQSjP 7exA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:subject:to:cc:references:in-reply-to :mime-version:date:message-id; bh=oYcZFNj7C4gqFVxy8ocUWkgd0xBG/Pe8ToC0wSpkruM=; b=mIQ6D5XKzGuCMbAzRQ2LvMj7gMpG3PdmoF0+5R9HV0yzWDYm2fXJ0bNhtrlT79rwj1 b7R8tLGTQx89mckWAKHgen4bmzjk6gXTjqd1MdBAbwtMnoXPoijzjCn23qSj5tHtGGA3 WbLsQkzwrGwn/6kw7IawVFN8kpGTBZ25dJQHxSBPQm32dZeZsmBgWU0NlgCVp+45j96j EMVZoUHqXSaAVcXxbwRxvgvbggFrqlA00xOtwSIvcwNzqpezqQwkbRDIe3yk5pzPRf9B 0sGn3SkWNNpgYJf+iIChsA/54G7uafR6wfm9DJLE+tvLPby8BSTxiiZfKH1m+BL1gGN7 h0Vg== X-Gm-Message-State: AGi0PuYLPsSRn9k47rZSatllDXiH+Bh/nYEdBdawqDHQuVOIkGYQo8D5 1y9NUdp22NTpRP0J5HGZ7lY= X-Google-Smtp-Source: APiQypJCJnkzLiKWd5uThYH9ALD+HZUAC/7dd2Q0g/7P+cKpgyX+mhbgUm9y7BDp+5PptFFiHI1YNw== X-Received: by 2002:adf:edcc:: with SMTP id v12mr12888406wro.317.1587753353700; Fri, 24 Apr 2020 11:35:53 -0700 (PDT) Received: from localhost ([79.116.186.159]) by smtp.gmail.com with ESMTPSA id y18sm4450598wmc.45.2020.04.24.11.35.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 Apr 2020 11:35:52 -0700 (PDT) From: Franz Fellner Subject: Re: performance problems with notmuch new To: Olly Betts References: <20200415150801.h2mazyo37sspvech@redhat.com> <1587211167-ner-6.432@LappyL520> <87imhup6kr.fsf@tethera.net> <20200422232130.GH28897@survex.com> In-Reply-To: <20200422232130.GH28897@survex.com> MIME-Version: 1.0 Date: Fri, 24 Apr 2020 21:35:55 +0300 Message-Id: <1587753355-ner-5.891@LappyL520> X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: notmuch@notmuchmail.org, Xapian Discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: notmuch-bounces@notmuchmail.org Sender: "notmuch" X-Scanner: scn0 X-Spam-Score: 0.59 Authentication-Results: aspmx1.migadu.com; dkim=fail (body hash did not verify) header.d=gmail.com header.s=20161025 header.b=krEqc3wE; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 50.126.95.6 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Scan-Result: default: False [0.59 / 13.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; GENERIC_REPUTATION(0.00)[-0.46246596081979]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+a:c]; R_DKIM_REJECT(1.00)[gmail.com:s=20161025]; DWL_DNSWL_BLOCKED(0.00)[50.126.95.6:from]; FREEMAIL_FROM(0.00)[gmail.com]; IP_REPUTATION_HAM(0.00)[asn: 27017(-0.18), country: US(-0.00), ip: 50.126.95.6(-0.46)]; DKIM_TRACE(0.00)[gmail.com:-]; MX_GOOD(-0.50)[cached: notmuchmail.org]; MAILLIST(-0.20)[mailman]; RECEIVED_SPAMHAUS_PBL(0.00)[79.116.186.159:received]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:27017, ipnet:50.126.64.0/18, country:US]; TAGGED_FROM(0.00)[]; FROM_NEQ_ENVFROM(0.00)[alpineartde@gmail.com,notmuch-bounces@notmuchmail.org]; ARC_NA(0.00)[]; URIBL_BLOCKED(0.00)[survex.com:email,notmuchmail.org:email]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[notmuch@notmuchmail.org]; HAS_LIST_UNSUB(-0.01)[]; DNSWL_BLOCKED(0.00)[50.126.95.6:from]; MID_RHS_NOT_FQDN(0.50)[]; RCVD_COUNT_SEVEN(0.00)[9]; FORGED_SENDER_MAILLIST(0.00)[]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : SPF not aligned (relaxed),none] X-TUID: vxqDb8JE8+X7 On Thu Apr 23 00:21:30 2020, Olly Betts wrote: > First question: what version of Xapian are you using? On my laptop it's 1.4.15 (arch linux) and the desktop runs 1.4.14 (Gentoo linux) > And second thing to check, are you committing each message separately? No, I sync with mbsync which dosnloads a bunch of mails, then I run notmuch new which indexes all in one go. > After reboot the disk cache won't have any of the database in, so the > first operation will typically be slower, especially with a spinning > drive where seeks are relatively slow. Yes, I know that, I just wanted to mention the number, which IMO is insane. I want to setup notmuch for my dad on the desktop PC. 5 minutes to wait for his mail in the morning would have made notmuch a no-go. > It sounds like you're seek-limited in this "cold cache" phase. That is > not necessarily related to the slow indexing, but it could be. > > I'd check the SMART diagnostics for the drive first (e.g. with > smartctl). It's not the most likely cause, but it's quick to check and > if the drive is starting to fail it's better to find out sooner rather > than later. HDDs are healthy. I actually checked quite recently when converting the laptop from Gentoo to arch. > > Then I'd try compacting the database (I think there's a "notmuch > compact" subcommand to do this). And there we go. Cured the issues. Dropped the very first indexing from several minutes to 1.5 seconds on the desktop. ?!?! This is a really new setup and I suffered from bad performance from the very first notmuch new after the initial indexing. Is it really needed to run notmch compact directly after the initial notmuch new? Desktop currently has 38502 messages indexed, in case that matters. Regards Franz