From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id 6FWSFq3rml5AaAAA0tVLHw (envelope-from ) for ; Sat, 18 Apr 2020 11:59:41 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id KGTBILHrml7rHQAAB5/wlQ (envelope-from ) for ; Sat, 18 Apr 2020 11:59:45 +0000 Received: from arlo.cworth.org (arlo.cworth.org [50.126.95.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 02225941DD9 for ; Sat, 18 Apr 2020 11:59:43 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 5A97A6DE105F; Sat, 18 Apr 2020 04:59:40 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id aa8D_9qXUdjD; Sat, 18 Apr 2020 04:59:39 -0700 (PDT) Received: from arlo.cworth.org (localhost [IPv6:::1]) by arlo.cworth.org (Postfix) with ESMTP id 13CDB6DE10B3; Sat, 18 Apr 2020 04:59:39 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 11AD96DE10B3 for ; Sat, 18 Apr 2020 04:59:38 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id X0GI3F8f4qWO for ; Sat, 18 Apr 2020 04:59:36 -0700 (PDT) Received: from mail-wr1-f43.google.com (mail-wr1-f43.google.com [209.85.221.43]) by arlo.cworth.org (Postfix) with ESMTPS id 582326DE105F for ; Sat, 18 Apr 2020 04:59:36 -0700 (PDT) Received: by mail-wr1-f43.google.com with SMTP id k11so6012403wrp.5 for ; Sat, 18 Apr 2020 04:59:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:references:in-reply-to:mime-version:date:message-id; bh=GxepQUf0Z8b1Gh6lTRK177di0sQtgLS5IaGOjog/GaM=; b=aCiCesz+skTuG0s355PmLHQAmD8gnuXFY0SKAF3Z8rpRRkljIf7v+de28ywFksMpKR UPnYl4EPKje+4n2XpbLykULrLklMdKQ6yVOrDGkjcRwo8vLoQwibJr2Qh/V4vuHUxv7M DRFCOKXuXmV1Fbg5W5usem4lQ+1psy4TkCMwKxzbgC2ht+CjWmgT89MIj196Lkz5icdx rLylAGeTmqeUlvFYxeKgJ5rH7kx/1A3L7oCTTnYTrye7X7NzwXY8HKpK+hiZQ59AbSdv suewEyeMTmbtub6HR+oDCUvGPjDJvsBxclVutvqONNI6cPdGfVKxe+Bupj1FOv0I1KtS 8JCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:references:in-reply-to :mime-version:date:message-id; bh=GxepQUf0Z8b1Gh6lTRK177di0sQtgLS5IaGOjog/GaM=; b=aDBW8C4tm+7/ULKQL1S9DilIFG+6W2FxlgsONgJ1euYEsRBN0yfQzMALggsh/xDUBF xqS1GV83xiZBnTNt1/hOdVuVTYVQZFCoFHGVM9oH0dMvHVzAqzew+cUb4PIuJF+UBtgb /PCiU/ldQFLnqSQRIl6Ko8zI99nhZo8FRxXbIBY3aQRPBKQvDt7xE5Ikm4edgkLmg0nX 452cpIV+AQPDE2SbH5PDZSwdgqVj1MqikPX550YoC25hku1jtsB5bHnKYaC53mGenpnC Lo9Q9oLtebuG1mevzfLNM1/VLtUJb9qnjOkEqvPLR1Le+2dfUeQWIrBUzB23excZhbKO 5NMA== X-Gm-Message-State: AGi0PuZHjxDlfa40ZCaPz61JkAUDl/5J2uh49tcZPZo1iLyySnSAK2zH sfCp5m8GMXCgqJMV7gXAtUs= X-Google-Smtp-Source: APiQypJ2qL18PQG/dLMgt129u7i43LY4+0DnKytWoQ3mePHvkRasYEWaAUJd/NPXHL6/vqeTjJEyxA== X-Received: by 2002:a5d:4dcb:: with SMTP id f11mr8607400wru.174.1587211174031; Sat, 18 Apr 2020 04:59:34 -0700 (PDT) Received: from localhost ([79.116.186.159]) by smtp.gmail.com with ESMTPSA id w12sm21263253wrk.56.2020.04.18.04.59.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 18 Apr 2020 04:59:33 -0700 (PDT) Subject: Re: performance problems with notmuch new From: Franz Fellner To: Don Zickus , notmuch@notmuchmail.org References: <20200415150801.h2mazyo37sspvech@redhat.com> In-Reply-To: <20200415150801.h2mazyo37sspvech@redhat.com> MIME-Version: 1.0 Date: Sat, 18 Apr 2020 14:59:27 +0300 Message-Id: <1587211167-ner-6.432@LappyL520> X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: notmuch-bounces@notmuchmail.org Sender: "notmuch" X-Scanner: scn0 X-Spam-Score: 1.89 Authentication-Results: aspmx1.migadu.com; dkim=fail (body hash did not verify) header.d=gmail.com header.s=20161025 header.b=aCiCesz+; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 50.126.95.6 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Scan-Result: default: False [1.89 / 13.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; GENERIC_REPUTATION(0.00)[-0.45642113552296]; MX_INVALID(1.00)[]; TO_DN_SOME(0.00)[]; R_DKIM_REJECT(1.00)[gmail.com:s=20161025]; FREEMAIL_FROM(0.00)[gmail.com]; IP_REPUTATION_HAM(0.00)[asn: 27017(-0.18), country: US(-0.01), ip: 50.126.95.6(-0.46)]; R_SPF_ALLOW(-0.20)[+a]; DKIM_TRACE(0.00)[gmail.com:-]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_MED(-0.20)[50.126.95.6:from]; MAILLIST(-0.20)[mailman]; RECEIVED_SPAMHAUS_PBL(0.00)[79.116.186.159:received]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:27017, ipnet:50.126.64.0/18, country:US]; TAGGED_FROM(0.00)[]; FROM_NEQ_ENVFROM(0.00)[alpineartde@gmail.com,notmuch-bounces@notmuchmail.org]; ARC_NA(0.00)[]; URIBL_BLOCKED(0.00)[notmuchmail.org:email]; FROM_HAS_DN(0.00)[]; SPF_REPUTATION_HAM(0.00)[-0.45616644824971]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[notmuch@notmuchmail.org]; HAS_LIST_UNSUB(-0.01)[]; MID_RHS_NOT_FQDN(0.50)[]; RCVD_COUNT_SEVEN(0.00)[9]; FORGED_SENDER_MAILLIST(0.00)[]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : SPF not aligned (relaxed),none] X-TUID: 6JrkHow/9210 I also suffer from bad performance of notmuch new. I used notmuch some years ago and notmuch new always felt instantanious. Had to stop using it because internet was too slow to sync my mails :/ Now (with better internet and a completely new setup using mbsync) indexing one mail takes at least 10 seconds, sometimes even more. It can go into minutes when I get lots of mail (~30...). When I run it after a reboot I can have breakfast while notmuch starts up... This is all on spinning rust. I thought of getting an SSD but not in the near future. What I observe during that time: notmuch doesn't really need much CPU. iotop shows constant read and write with extremely low rates, under 1MB/sec. So I think it might be an issue in xapian? If there is anything I can do to help debug this please tell me Franz P.S.: @David: Sorry for writing only to you. GMail web interface only added you as recipient and not the list... On Wed Apr 15 11:08:01 2020, Don Zickus wrote: > Hi, > > I have noticed my 'notmuch new' command seems awfully slow, maybe 10-20 > emails / second on large batches. It goes quick for the first few hundred > (maybe close to 100/second), then quickly slows down to about 10/second > after processing the first 500 or so. > > I am guessing that isn't an expected behaviour. So I am trying to figure > out a good way to analyze and debug this? This could be a problem with my > fedora distro or laptop. I just don't know where to look. > > Tips for debugging this? > > I ran the notmuch performance/time-test, but after 15 minutes of waiting for > the initial notmuch new to finish, I gave up and aborted. > > I am using 'glass' for my xapian storage if that helps. > > Help? > > Cheers, > Don > > _______________________________________________ > notmuch mailing list > notmuch@notmuchmail.org > https://notmuchmail.org/mailman/listinfo/notmuch > > --