From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id 2P3cOLs3CWJUGAAAgWs5BA (envelope-from ) for ; Sun, 13 Feb 2022 17:54:19 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id OBysMbs3CWLRHAAAG6o9tA (envelope-from ) for ; Sun, 13 Feb 2022 17:54:19 +0100 Received: from mail.notmuchmail.org (yantan.tethera.net [IPv6:2a01:4f9:c011:7a79::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 2676713FF5 for ; Sun, 13 Feb 2022 17:54:19 +0100 (CET) Received: from yantan.tethera.net (localhost [127.0.0.1]) by mail.notmuchmail.org (Postfix) with ESMTP id 031D95F6C1; Sun, 13 Feb 2022 16:54:17 +0000 (UTC) Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) by mail.notmuchmail.org (Postfix) with ESMTPS id 1D3225F401 for ; Sun, 13 Feb 2022 16:54:14 +0000 (UTC) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id 7FEC35C0161; Sun, 13 Feb 2022 11:54:12 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Sun, 13 Feb 2022 11:54:12 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=spwhitton.name; h=cc:cc:content-transfer-encoding:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to; s=fm2; bh=wM05NZjcwXSLCF mlzO0EZAuQNjZ2cMgoBKr0Fk21YAc=; b=kw8qkYS0JVsjowGZBrbtHm+I5uTICo xsrws30/CRo8E1bnyb4gbf0/0A8jo+bnfTjU50xqo1fGq/wY61pi1vP2I39L4c/N vgDdEMmi2sLCEV5PewiXpLaza+xg2mn7OGCckzWQX1P2JQtTRcW5EJEQzRCuUOn7 /c6PRdx2OsEZ6qW7nlMbtHh24C8jrpFi7owK5mcO/440QFXShycGP2whE2A8nnJ8 GF4ZtHOLcgSISlcOKqij38V8aj66nAx6nyiUnSa7CJZ1wlGvy7ol/LVINhGoWEqP KMYw2tJ7r8iFQUrAb56hihjJTkFIa0Gv/i2L8/c8FZts16guy6Z07WyQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding:date:date :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=wM05NZ jcwXSLCFmlzO0EZAuQNjZ2cMgoBKr0Fk21YAc=; b=ScuCRjdxYPLm+dgRkX5ilG oGguTCUXJFtXgPcC405emTUIgO8NuCR8+Spg6J9YIwT6PvFvIyV3+HCKiNMyb8MZ R31TdIKONec8IBM08qddfYwEByK3Zu3Cj7gkPf7NSW4yY/luaTMVN+J+YnOIckqB ZG1nW6h7ryE1aJN9DAddYqJwn1+4b2OTmmD2p63WuwLJphI9ryegGnT+L/yEaGIA V9WfbvlYddrMS6lR/2lNEWrG+VhHtOSXwjYIsiO9K7n2Pu4Tf3meeQBeKytKacvK WzXtHccQSv9bczCpMg4DlKJfbAieVe5nVbMPyT8LJc5IW4DSqf404xoIyDvNYn/A == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrjedtgdeltdcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhephffvufffkffojghfggfgsedtkeertd ertddtnecuhfhrohhmpefuvggrnhcuhghhihhtthhonhcuoehsphifhhhithhtohhnsehs phifhhhithhtohhnrdhnrghmvgeqnecuggftrfgrthhtvghrnheptefhleetteeuiedthe elhffguddvffdthfeiffetteefkedvheeltedvjeffuedunecuvehluhhsthgvrhfuihii vgeptdenucfrrghrrghmpehmrghilhhfrhhomhepshhpfihhihhtthhonhesshhpfihhih htthhonhdrnhgrmhgv X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 13 Feb 2022 11:54:12 -0500 (EST) Received: by athena.silentflame.com (Postfix, from userid 1000) id D74801B58E6; Sun, 13 Feb 2022 16:54:10 +0000 (UTC) From: Sean Whitton To: notmuch@notmuchmail.org Subject: [PATCH v2] nmbug: write tags out to a temporary file, not 'nmbug.index' Date: Sun, 13 Feb 2022 09:54:08 -0700 Message-Id: <20220213165408.972859-1-spwhitton@spwhitton.name> X-Mailer: git-send-email 2.30.2 In-Reply-To: References: MIME-Version: 1.0 Message-ID-Hash: A2A6AFZMKGXG6LVNQNR57HBKTMV4BR4C X-Message-ID-Hash: A2A6AFZMKGXG6LVNQNR57HBKTMV4BR4C X-MailFrom: spwhitton@spwhitton.name X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-notmuch.notmuchmail.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Sean Whitton X-Mailman-Version: 3.3.3 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Migadu-Flow: FLOW_IN X-Migadu-Country: DE ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1644771259; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-owner:list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=kpCmaUNb85/V8xmm43jwUCb46hKRIykQFIU62t36To4=; b=YSo4C4zGoPsgvNok0LYsDk3QJr9q7snl8gvN+O7EAkd3bTbpC7AczmUzH8BaWsw64LE1ro EAh3x1ZgvUgQOS7gvPTT/A5SGf9V5/sXw48UQrdoG2sRv09OZR8FLrFMjaSkvS37fSe/VU 1kslbgJcsE1WSRhREBdtLk1dI7dwx/UKYC44o6CT3n3p/xZyecl44Sfqy2eJ1Rsj2S4kZI R77JqSwPl4GJaGSlMjWNmxzuWb7m4rVngF6fph0lnYnaVmgtD4Rj73+tGUKhOoSFPhomi8 ww5Qw/eUgBPPZ6gNG4l81XBeV9OuTp0im36hfI7YlrXzBjTfaV4pkEIgVYm97g== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1644771259; a=rsa-sha256; cv=none; b=S49X6Q1EYl62n22/0UFn8lUZwnc8LJJcCXXY6nSsXe/8Pd6QZr7WxHVVH+br1063sxMmow Jc9lohEZQKM69eYa/s498iHaeT8/aTqDBV/wi2zf+jLJH0wTO/3+w6p7d3RicEla8DpfGD i9OQ1YZPt1B2D6gWngWe4gzHbT4rZKxbjVYAyhkz9HPOCNXIzfR8TS+s8ug7XlhxhyNeJs F+Ur++R+uO3y8KO9ccemOESm937Rx5ktoe+YFnt8xlp4GM9y5WKGwDU5ZzhsZoePSj4mM5 d3fOiJqNy8401nNNI1bBJZHzEWtnR0wv1Xr72GkCyldM0ZSl4NhPXMfM/fMmtQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=spwhitton.name header.s=fm2 header.b=kw8qkYS0; dkim=fail ("body hash did not verify") header.d=messagingengine.com header.s=fm2 header.b=ScuCRjdx; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 2a01:4f9:c011:7a79::1 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Spam-Score: -0.44 Authentication-Results: aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=spwhitton.name header.s=fm2 header.b=kw8qkYS0; dkim=fail ("body hash did not verify") header.d=messagingengine.com header.s=fm2 header.b=ScuCRjdx; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 2a01:4f9:c011:7a79::1 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Queue-Id: 2676713FF5 X-Spam-Score: -0.44 X-Migadu-Scanner: scn1.migadu.com X-TUID: stuqPlNcjpeR If more than nmbug process is running at once, then each will try to read and write the same file. The particular failure I've seen is that the process which finishes first deletes nmbug.index, and then the other process dies with a FileNotFoundError. So use a distinct temporary file per process. --- devel/nmbug/nmbug | 44 ++++++++++++++++++++++---------------------- 1 file changed, 22 insertions(+), 22 deletions(-) Here is a second attempt, though I'm afraid I have little idea whether it is idiomatic Python. diff --git a/devel/nmbug/nmbug b/devel/nmbug/nmbug index 043c1863..396098d0 100755 --- a/devel/nmbug/nmbug +++ b/devel/nmbug/nmbug @@ -586,37 +586,38 @@ def get_status(): 'deleted': {}, 'missing': {}, } - index = _index_tags() - maybe_deleted = _diff_index(index=index, filter='D') - for id, tags in maybe_deleted.items(): - (_, stdout, stderr) = _spawn( - args=['notmuch', 'search', '--output=files', 'id:{0}'.format(id)], - stdout=_subprocess.PIPE, - wait=True) - if stdout: - status['deleted'][id] = tags - else: - status['missing'][id] = tags - status['added'] = _diff_index(index=index, filter='A') - _os.remove(index) - return status - - -def _index_tags(): - "Write notmuch tags to the nmbug.index." - path = _os.path.join(NMBGIT, 'nmbug.index') + (_, index) = _tempfile.mkstemp() + try: + _index_tags(index) + maybe_deleted = _diff_index(index=index, filter='D') + for id, tags in maybe_deleted.items(): + (_, stdout, stderr) = _spawn( + args=['notmuch', 'search', '--output=files', 'id:{0}'.format(id)], + stdout=_subprocess.PIPE, + wait=True) + if stdout: + status['deleted'][id] = tags + else: + status['missing'][id] = tags + status['added'] = _diff_index(index=index, filter='A') + return status + finally: + _os.remove(index) + +def _index_tags(index): + "Write notmuch tags to a file." query = ' '.join('tag:"{tag}"'.format(tag=tag) for tag in get_tags()) prefix = '+{0}'.format(_ENCODED_TAG_PREFIX) _git( args=['read-tree', '--empty'], - additional_env={'GIT_INDEX_FILE': path}, wait=True) + additional_env={'GIT_INDEX_FILE': index}, wait=True) with _spawn( args=['notmuch', 'dump', '--format=batch-tag', '--', query], stdout=_subprocess.PIPE) as notmuch: with _git( args=['update-index', '--index-info'], stdin=_subprocess.PIPE, - additional_env={'GIT_INDEX_FILE': path}) as git: + additional_env={'GIT_INDEX_FILE': index}) as git: for line in notmuch.stdout: if line.strip().startswith('#'): continue @@ -629,7 +630,6 @@ def _index_tags(): for line in _index_tags_for_message( id=id, status='A', tags=tags): git.stdin.write(line) - return path def _index_tags_for_message(id, status, tags): -- 2.30.2