From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id uFNvKr8rK2TIdQAASxT56A (envelope-from ) for ; Mon, 03 Apr 2023 21:40:47 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id KNBzKr8rK2Q4OwAA9RJhRA (envelope-from ) for ; Mon, 03 Apr 2023 21:40:47 +0200 Received: from mail.notmuchmail.org (yantan.tethera.net [IPv6:2a01:4f9:c011:7a79::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 4638FE3E0 for ; Mon, 3 Apr 2023 21:40:47 +0200 (CEST) Received: from yantan.tethera.net (localhost [127.0.0.1]) by mail.notmuchmail.org (Postfix) with ESMTP id 055895F428; Mon, 3 Apr 2023 19:40:45 +0000 (UTC) Received: from fethera.tethera.net (fethera.tethera.net [IPv6:2607:5300:60:c5::1]) by mail.notmuchmail.org (Postfix) with ESMTP id 5CF1C5E53D for ; Mon, 3 Apr 2023 19:40:42 +0000 (UTC) Received: by fethera.tethera.net (Postfix, from userid 1001) id B1BCD5FB82; Mon, 3 Apr 2023 15:40:41 -0400 (EDT) Received: (nullmailer pid 11707 invoked by uid 1000); Mon, 03 Apr 2023 19:40:40 -0000 From: David Bremner To: Felipe Contreras Subject: Re: Reimagining notmuch-git/nmbug In-Reply-To: <87pm8kn6xf.fsf@minkowski.home> References: <871ql12t34.fsf@tethera.net> <87pm8kn6xf.fsf@minkowski.home> Date: Mon, 03 Apr 2023 16:40:40 -0300 Message-ID: <87mt3on493.fsf@minkowski.home> MIME-Version: 1.0 Message-ID-Hash: 67UACQP2OTBBCO2IOPVMERS4FT3VQAJT X-Message-ID-Hash: 67UACQP2OTBBCO2IOPVMERS4FT3VQAJT X-MailFrom: david@tethera.net X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-notmuch.notmuchmail.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: "notmuch@notmuchmail.org" X-Mailman-Version: 3.3.3 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Migadu-Country: DE X-Migadu-Flow: FLOW_IN ARC-Seal: i=1; s=key1; d=yhetil.org; t=1680550847; a=rsa-sha256; cv=none; b=aDKG7A3ai1ILsRxNWt7s6UibwGkNT6yVqKQYVLC9QS7IbTauLeAT6pmKIwBAcOjPdu4nr0 wMooAnXCxqyro7Qt+U/E17fg1AAZTMRQG4vwZsTOfoZf5LgNeoL0aooUne3Y0CqSLH4X3Y l168Ol7gvNfhOD9FntZ0A8lVlPPa8ncIUIAC+pLF+oLn7q8mkIXgYPGDQMK7a7ADt2iKQI K8QdRXykDEkGd144WmASzoNYp4oCX79g5mKxxpSQavr8Q9m0PXSWUafTuw8r2i/xd/lFX/ 4p7nEYF9mF/tv+5uxbU6cKpgVQLxarNYJF+vSve3NMBIuHPsxSbMVVyz79sRTA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 2a01:4f9:c011:7a79::1 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1680550847; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-owner:list-unsubscribe:list-subscribe:list-post; bh=rJRzGG+RpXqlZQFqXHV4Vsb3qqynTQywYPdGY7KYH7k=; b=Ho1UDBBIi+tft3EkWegLDJAUJ450rAR1A59VoNUb6En3DSnJE2+MN+kTl47lKPEO6wkJ54 x3rsQLQgpQ6bD7Kl8IDI5UH70auwjoyX55BqGj/upIeySX6NIzy2keYVXY6L74uC8nlEv6 CeO3qM7eLdGkdPJlJysUwbvd09mT8nyFpB/V6xZORRg5gaxyx5rv7VeMyUgX/6+iCWxkXB WRzyrdHfEJZ4UHqYzx93bdcTpL1MBhz+GRVCY1t3F9KHAFwrz/lHRYEK8e+FF2CJfRG4Al l7l5fjmwXWN6+QmalxGl2nQM4puWedWeqZ/ZyJoEy0gvZZDcGz2yMDmYm3D2lw== Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 2a01:4f9:c011:7a79::1 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Scanner: scn0.migadu.com X-Migadu-Spam-Score: -2.08 X-Spam-Score: -2.08 X-Migadu-Queue-Id: 4638FE3E0 X-TUID: wRp2CEp/IjCw David Bremner writes: > Indeed that speeds up the initial clone on this machine from 39 minutes > (I switched machines) to 30s. I will play with it a bit more, and report > back. It's not a showstopper, but "git pull" takes about 1/2 the wall time (about 2/3 of the CPU time) of the original clone, even if there is only one tag changed. Two potential improvements I can think of. - notmuch-dump.c calls notmuch_query_set_sort (query, NOTMUCH_SORT_UNSORTED). I think I managed to do this (diff below), but performance gain was negligible. - Since you cache the lastmod value, you should be able to use it in a query. This does make a big difference in my experiments. I had to remove the 'deleteall' (otherwise only the changed messages are left in the git repo). I'm not 100% this is correct, hopefully you see quicker than I. In any case the lastmod query is what notmuch-git uses. diff --git a/git-remote-nm b/git-remote-nm index c668b38..cabea26 100755 --- a/git-remote-nm +++ b/git-remote-nm @@ -148,9 +148,11 @@ def wr_import(ref) wr_data("lastmod: %d\n" % ($lastmod || 0)) wr_l 'from refs/notmuch/master^0' if $lastmod - wr_l 'deleteall' +# wr_l 'deleteall' - $db.query('').search_messages.each do |msg| + $query=$db.query("lastmod:%d.." % ($lastmod || 0) ) + $query.sort=Notmuch::SORT_UNSORTED + $query.search_messages.each do |msg| hash = Blake2b.hex(msg.message_id, Blake2b::Key.none, 2) dir1, dir2 = hash[..1], hash[2..] wr_l 'M 644 inline %s/%s/%s/tags' % [dir1, dir2, encode_filename(msg.message_id)]