* [PATCH v2] nmbug: write tags out to a temporary file, not 'nmbug.index' [not found] <id:m2pmnr2o2f.fsf@guru.guru-group.fi> @ 2022-02-13 16:54 ` Sean Whitton 2022-02-13 17:17 ` Kyle Meyer 2022-02-13 17:18 ` Sean Whitton 0 siblings, 2 replies; 4+ messages in thread From: Sean Whitton @ 2022-02-13 16:54 UTC (permalink / raw) To: notmuch; +Cc: Sean Whitton If more than nmbug process is running at once, then each will try to read and write the same file. The particular failure I've seen is that the process which finishes first deletes nmbug.index, and then the other process dies with a FileNotFoundError. So use a distinct temporary file per process. --- devel/nmbug/nmbug | 44 ++++++++++++++++++++++---------------------- 1 file changed, 22 insertions(+), 22 deletions(-) Here is a second attempt, though I'm afraid I have little idea whether it is idiomatic Python. diff --git a/devel/nmbug/nmbug b/devel/nmbug/nmbug index 043c1863..396098d0 100755 --- a/devel/nmbug/nmbug +++ b/devel/nmbug/nmbug @@ -586,37 +586,38 @@ def get_status(): 'deleted': {}, 'missing': {}, } - index = _index_tags() - maybe_deleted = _diff_index(index=index, filter='D') - for id, tags in maybe_deleted.items(): - (_, stdout, stderr) = _spawn( - args=['notmuch', 'search', '--output=files', 'id:{0}'.format(id)], - stdout=_subprocess.PIPE, - wait=True) - if stdout: - status['deleted'][id] = tags - else: - status['missing'][id] = tags - status['added'] = _diff_index(index=index, filter='A') - _os.remove(index) - return status - - -def _index_tags(): - "Write notmuch tags to the nmbug.index." - path = _os.path.join(NMBGIT, 'nmbug.index') + (_, index) = _tempfile.mkstemp() + try: + _index_tags(index) + maybe_deleted = _diff_index(index=index, filter='D') + for id, tags in maybe_deleted.items(): + (_, stdout, stderr) = _spawn( + args=['notmuch', 'search', '--output=files', 'id:{0}'.format(id)], + stdout=_subprocess.PIPE, + wait=True) + if stdout: + status['deleted'][id] = tags + else: + status['missing'][id] = tags + status['added'] = _diff_index(index=index, filter='A') + return status + finally: + _os.remove(index) + +def _index_tags(index): + "Write notmuch tags to a file." query = ' '.join('tag:"{tag}"'.format(tag=tag) for tag in get_tags()) prefix = '+{0}'.format(_ENCODED_TAG_PREFIX) _git( args=['read-tree', '--empty'], - additional_env={'GIT_INDEX_FILE': path}, wait=True) + additional_env={'GIT_INDEX_FILE': index}, wait=True) with _spawn( args=['notmuch', 'dump', '--format=batch-tag', '--', query], stdout=_subprocess.PIPE) as notmuch: with _git( args=['update-index', '--index-info'], stdin=_subprocess.PIPE, - additional_env={'GIT_INDEX_FILE': path}) as git: + additional_env={'GIT_INDEX_FILE': index}) as git: for line in notmuch.stdout: if line.strip().startswith('#'): continue @@ -629,7 +630,6 @@ def _index_tags(): for line in _index_tags_for_message( id=id, status='A', tags=tags): git.stdin.write(line) - return path def _index_tags_for_message(id, status, tags): -- 2.30.2 ^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH v2] nmbug: write tags out to a temporary file, not 'nmbug.index' 2022-02-13 16:54 ` [PATCH v2] nmbug: write tags out to a temporary file, not 'nmbug.index' Sean Whitton @ 2022-02-13 17:17 ` Kyle Meyer 2022-02-13 17:18 ` Sean Whitton 1 sibling, 0 replies; 4+ messages in thread From: Kyle Meyer @ 2022-02-13 17:17 UTC (permalink / raw) To: Sean Whitton; +Cc: notmuch [ drive-by comment based on a past mistake :/ ] Sean Whitton writes: > -def _index_tags(): > - "Write notmuch tags to the nmbug.index." > - path = _os.path.join(NMBGIT, 'nmbug.index') > + (_, index) = _tempfile.mkstemp() [...] > _git( > args=['read-tree', '--empty'], > - additional_env={'GIT_INDEX_FILE': path}, wait=True) > + additional_env={'GIT_INDEX_FILE': index}, wait=True) It's better to put the temporary index in $GIT_DIR due to this bit noted in git-read-tree(1): The file must allow to be rename(2)ed into from a temporary file that is created next to the usual index file; typically this means it needs to be on the same filesystem as the index file itself, and you need write permission to the directories the index file and index output file are located in. So, assuming NMBGIT matches $GIT_DIR, perhaps change the above mkstemp call to something like _tempfile.mkstemp(dir=NMBGIT, prefix="nmbug", suffix=".index") ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH v2] nmbug: write tags out to a temporary file, not 'nmbug.index' 2022-02-13 16:54 ` [PATCH v2] nmbug: write tags out to a temporary file, not 'nmbug.index' Sean Whitton 2022-02-13 17:17 ` Kyle Meyer @ 2022-02-13 17:18 ` Sean Whitton 2022-02-13 18:08 ` Tomi Ollila 1 sibling, 1 reply; 4+ messages in thread From: Sean Whitton @ 2022-02-13 17:18 UTC (permalink / raw) To: notmuch Hello, On Sun 13 Feb 2022 at 09:54am -07, Sean Whitton wrote: > If more than nmbug process is running at once, then each will try to > read and write the same file. The particular failure I've seen is > that the process which finishes first deletes nmbug.index, and then > the other process dies with a FileNotFoundError. So use a distinct > temporary file per process. > --- > devel/nmbug/nmbug | 44 ++++++++++++++++++++++---------------------- > 1 file changed, 22 insertions(+), 22 deletions(-) > > Here is a second attempt, though I'm afraid I have little idea whether it is > idiomatic Python. It would seem this causes 'nmbug status' to output just one result. I'll leave the fix to someone with more Python experience. Sorry for the improperly tested v2 patch. -- Sean Whitton ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH v2] nmbug: write tags out to a temporary file, not 'nmbug.index' 2022-02-13 17:18 ` Sean Whitton @ 2022-02-13 18:08 ` Tomi Ollila 0 siblings, 0 replies; 4+ messages in thread From: Tomi Ollila @ 2022-02-13 18:08 UTC (permalink / raw) To: Sean Whitton, notmuch On Sun, Feb 13 2022, Sean Whitton wrote: > Hello, > > On Sun 13 Feb 2022 at 09:54am -07, Sean Whitton wrote: > >> If more than nmbug process is running at once, then each will try to >> read and write the same file. The particular failure I've seen is >> that the process which finishes first deletes nmbug.index, and then >> the other process dies with a FileNotFoundError. So use a distinct >> temporary file per process. >> --- >> devel/nmbug/nmbug | 44 ++++++++++++++++++++++---------------------- >> 1 file changed, 22 insertions(+), 22 deletions(-) >> >> Here is a second attempt, though I'm afraid I have little idea whether it is >> idiomatic Python. > > It would seem this causes 'nmbug status' to output just one result. > > I'll leave the fix to someone with more Python experience. Sorry for > the improperly tested v2 patch. One option would be using the first patch, but instead of mkstemp(), NamedTemporaryFile(dir=NMBGIT, prefix="nmbug.index") to be used instead (and then path.name and index.name in place of path and index...) Tomi > > -- > Sean Whitton ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2022-02-13 18:09 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <id:m2pmnr2o2f.fsf@guru.guru-group.fi> 2022-02-13 16:54 ` [PATCH v2] nmbug: write tags out to a temporary file, not 'nmbug.index' Sean Whitton 2022-02-13 17:17 ` Kyle Meyer 2022-02-13 17:18 ` Sean Whitton 2022-02-13 18:08 ` Tomi Ollila
Code repositories for project(s) associated with this public inbox https://yhetil.org/notmuch.git/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).