From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id kC2LG6OxwWJLRQAAbAwnHQ (envelope-from ) for ; Sun, 03 Jul 2022 17:11:31 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id aOlRG6OxwWLlEwEAauVa8A (envelope-from ) for ; Sun, 03 Jul 2022 17:11:31 +0200 Received: from mail.notmuchmail.org (yantan.tethera.net [IPv6:2a01:4f9:c011:7a79::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 42C34C5E5 for ; Sun, 3 Jul 2022 17:11:31 +0200 (CEST) Received: from yantan.tethera.net (localhost [127.0.0.1]) by mail.notmuchmail.org (Postfix) with ESMTP id 5A5815E226; Sun, 3 Jul 2022 15:11:17 +0000 (UTC) Received: from fethera.tethera.net (fethera.tethera.net [198.245.60.197]) by mail.notmuchmail.org (Postfix) with ESMTP id 73E1A5E226 for ; Sun, 3 Jul 2022 15:11:14 +0000 (UTC) Received: by fethera.tethera.net (Postfix, from userid 1001) id B791D5FBD0; Sun, 3 Jul 2022 11:11:13 -0400 (EDT) Received: (nullmailer pid 1800794 invoked by uid 1000); Sun, 03 Jul 2022 15:11:09 -0000 From: David Bremner To: notmuch@notmuchmail.org Subject: [PATCH 3/4] CLI/git: current cache contents (file list) of index Date: Sun, 3 Jul 2022 12:11:02 -0300 Message-Id: <20220703151103.1800726-4-david@tethera.net> X-Mailer: git-send-email 2.35.2 In-Reply-To: <20220703151103.1800726-1-david@tethera.net> References: <20220703151103.1800726-1-david@tethera.net> MIME-Version: 1.0 Message-ID-Hash: WCKGHLXX4OH7TC43W7TWEIZ6L5MOR7MC X-Message-ID-Hash: WCKGHLXX4OH7TC43W7TWEIZ6L5MOR7MC X-MailFrom: bremner@tethera.net X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-notmuch.notmuchmail.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.3 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Migadu-Flow: FLOW_IN X-Migadu-To: larch@yhetil.org X-Migadu-Country: DE ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1656861091; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-owner:list-unsubscribe:list-subscribe:list-post; bh=ivHS80VMmIGQgZ1wTu3O4IsN1LcEskuQQ4lNap5u/Qs=; b=PwY2Nm+sVyij3ZyxbiWnf8tNPOVIbFWxvHSvwkrVuZzGvGbio2c8EfUjX1Ww6DjZ3rLqcP AvPzJ1/SLVE5YFSHx61/May4DWd1MLcmNQMpoA04X3+TvSX66o/242AmlEYBXuZoq8Dhsz MlfhOQBYEUnIXC0dLyvdKWgYC8mEbi1qs6qaP5kdmxp/uKwCnTPN5AX4B4IGvHwtNCV6yT whILG+7m3xH5LfO5rpHKfq5dl0TMaGej79IKSDMW5GgP3I26v3KIV1gzw+u4i5Hww0HMwJ OOxrLe4LhDC7mZz/oZ4f5MSfPmFxr8mfRrZBR9na5J4e1TYlc+2FaPzCWcs0mw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1656861091; a=rsa-sha256; cv=none; b=e3y6fz1Z76G6K6vDXrq0aDVG17T9DO0gOOh2+9HLYvKsr2FsdB2u891xVCh6lwiC73hp6Q bjx+zqiN3YrIsBFkYMdiEtTv7YE6PyyvoeLHSoNLsirh4HI1bVqLm5aedfSgIFF2L8lH4i DrBpmapbP5DtDob1eXm7GNP9yRVqSyNennlKV0OQ7cwIDEVeDJwuXK6zy3us0k2R4lDuwI fYaWOTZ2zw4m/q4c9GRMaVYyjXOKO25Am5lYcbxfsjVL2fObv3shitRbYFWoDDvUPKeBUk el3SY5FWsiNfqcefNqkiekTFD7fBnfdRWj7LfSWeuwsLqMw9chsYXMtKusA1NQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 2a01:4f9:c011:7a79::1 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Spam-Score: -1.41 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 2a01:4f9:c011:7a79::1 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Queue-Id: 42C34C5E5 X-Spam-Score: -1.41 X-Migadu-Scanner: scn0.migadu.com X-TUID: fqLs9tQX1mKx Rather than shelling out once per message to get the list of files corresponding to tags, it is much faster (although potentially a bit memory intensive) to read them all at once. --- notmuch-git.py | 58 +++++++++++++++++++++++++++++++++----------------- 1 file changed, 39 insertions(+), 19 deletions(-) diff --git a/notmuch-git.py b/notmuch-git.py index b3ae044e..a3ae15f7 100644 --- a/notmuch-git.py +++ b/notmuch-git.py @@ -738,6 +738,7 @@ class PrivateIndex: self.lastmod = None self.checksum = None self._load_cache_file() + self.file_tree = None self._index_tags() def __enter__(self): @@ -763,6 +764,43 @@ class PrivateIndex: _LOG.error("Error decoding cache") _sys.exit(1) + @timed + def _read_file_tree(self): + self.file_tree = {} + + with _git( + args = ['ls-files', 'tags'], + additional_env = {'GIT_INDEX_FILE': self.index_path}, + stdout = _subprocess.PIPE) as git: + for file in git.stdout: + dir = _os.path.dirname(file) + tag = _os.path.basename(file).rstrip() + if dir not in self.file_tree: + self.file_tree[dir] = [tag] + else: + self.file_tree[dir].append(tag) + + + def _clear_tags_for_message(self, id): + """ + Clear any existing index entries for message 'id' + + Neither 'id' nor the tags in 'tags' should be encoded/escaped. + """ + + if self.file_tree == None: + self._read_file_tree() + + dir = _id_path(id) + + if dir not in self.file_tree: + return + + for file in self.file_tree[dir]: + line = '0 0000000000000000000000000000000000000000\t{:s}/{:s}\n'.format(dir,file) + yield line + + @timed def _index_tags(self): "Write notmuch tags to private git index." @@ -798,7 +836,7 @@ class PrivateIndex: if tag.startswith(prefix)] id = _xapian_unquote(string=id) if clear_tags: - for line in _clear_tags_for_message(index=self.index_path, id=id): + for line in self._clear_tags_for_message(id=id): git.stdin.write(line) for line in _index_tags_for_message( id=id, status='A', tags=tags): @@ -835,24 +873,6 @@ def _read_index_checksum (index_path): except FileNotFoundError: return None - -def _clear_tags_for_message(index, id): - """ - Clear any existing index entries for message 'id' - - Neither 'id' nor the tags in 'tags' should be encoded/escaped. - """ - - dir = _id_path(id) - - with _git( - args=['ls-files', dir], - additional_env={'GIT_INDEX_FILE': index}, - stdout=_subprocess.PIPE) as git: - for file in git.stdout: - line = '0 0000000000000000000000000000000000000000\t{:s}\n'.format(file.strip()) - yield line - def _read_database_lastmod(): with _spawn( args=['notmuch', 'count', '--lastmod', '*'], -- 2.35.2