* New Python bindings (notmuch2 module) fail to exclude tags
@ 2020-11-20 15:54 Jorge P. de Morais Neto
2020-11-20 18:02 ` David Bremner
2020-11-20 18:17 ` Floris Bruynooghe
0 siblings, 2 replies; 6+ messages in thread
From: Jorge P. de Morais Neto @ 2020-11-20 15:54 UTC (permalink / raw)
To: notmuch
Hi. I am trying to migrate my Python3 script to the new Python bindings
(notmuch2 module). However, I cannot obtain a count of messages
matching a query excluding messages that have an exclude tag. From the
command line:
$ notmuch count 'is:.bf_spam'
0
The CLI command correctly counts zero messages having '.bf_spam' tag,
because all such messages also have the excluded 'spam' tag. But from
Python:
$ python3
Python 3.7.3 (default, Jul 25 2020, 13:03:44)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import notmuch2
>>> nm_db=notmuch2.Database()
>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',))
379
>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',), omit_excluded=nm_db.EXCLUDE.FALSE)
379
>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',), omit_excluded=nm_db.EXCLUDE.TRUE)
379
>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',), omit_excluded=nm_db.EXCLUDE.ALL)
379
>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',), omit_excluded=nm_db.EXCLUDE.FLAG)
379
Am I doing something wrong?
Regards
--
- <https://jorgemorais.gitlab.io/justice-for-rms/>
- If an email of mine arrives at your spam box, please notify me.
- Please adopt free/libre formats like PDF, ODF, Org, LaTeX, Opus, WebM and 7z.
- Free/libre software for Replicant, LineageOS and Android: https://f-droid.org
- [[https://www.gnu.org/philosophy/free-sw.html][What is free software?]]
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: New Python bindings (notmuch2 module) fail to exclude tags
2020-11-20 15:54 New Python bindings (notmuch2 module) fail to exclude tags Jorge P. de Morais Neto
@ 2020-11-20 18:02 ` David Bremner
2020-11-20 18:17 ` Floris Bruynooghe
1 sibling, 0 replies; 6+ messages in thread
From: David Bremner @ 2020-11-20 18:02 UTC (permalink / raw)
To: Jorge P. de Morais Neto, notmuch; +Cc: Floris Bruynooghe
"Jorge P. de Morais Neto" <jorge+list@disroot.org> writes:
> Hi. I am trying to migrate my Python3 script to the new Python bindings
> (notmuch2 module). However, I cannot obtain a count of messages
> matching a query excluding messages that have an exclude tag. From the
> command line:
> $ notmuch count 'is:.bf_spam'
> 0
>
> The CLI command correctly counts zero messages having '.bf_spam' tag,
> because all such messages also have the excluded 'spam' tag. But from
> Python:
>
> $ python3
> Python 3.7.3 (default, Jul 25 2020, 13:03:44)
> [GCC 8.3.0] on linux
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import notmuch2
>>>> nm_db=notmuch2.Database()
>>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',))
> 379
>>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',), omit_excluded=nm_db.EXCLUDE.FALSE)
> 379
>>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',), omit_excluded=nm_db.EXCLUDE.TRUE)
> 379
>>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',), omit_excluded=nm_db.EXCLUDE.ALL)
> 379
>>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',), omit_excluded=nm_db.EXCLUDE.FLAG)
> 379
>
It looks like a bug, but I'm not very expert with the python bindings,
so I've CCed the author of the new bindings. It would be great if you
could make a small test case that does not rely on your mail store.
d
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: New Python bindings (notmuch2 module) fail to exclude tags
2020-11-20 15:54 New Python bindings (notmuch2 module) fail to exclude tags Jorge P. de Morais Neto
2020-11-20 18:02 ` David Bremner
@ 2020-11-20 18:17 ` Floris Bruynooghe
2020-11-21 21:51 ` Jorge P. de Morais Neto
1 sibling, 1 reply; 6+ messages in thread
From: Floris Bruynooghe @ 2020-11-20 18:17 UTC (permalink / raw)
To: Jorge P. de Morais Neto, notmuch
Hi Jorge,
On Fri 20 Nov 2020 at 12:54 -0300, Jorge P. de Morais Neto wrote:
> Hi. I am trying to migrate my Python3 script to the new Python bindings
> (notmuch2 module). However, I cannot obtain a count of messages
> matching a query excluding messages that have an exclude tag. From the
> command line:
> $ notmuch count 'is:.bf_spam'
> 0
>
> The CLI command correctly counts zero messages having '.bf_spam' tag,
> because all such messages also have the excluded 'spam' tag. But from
> Python:
>
> $ python3
> Python 3.7.3 (default, Jul 25 2020, 13:03:44)
> [GCC 8.3.0] on linux
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import notmuch2
>>>> nm_db=notmuch2.Database()
>>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',))
> 379
>>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',), omit_excluded=nm_db.EXCLUDE.FALSE)
> 379
>>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',), omit_excluded=nm_db.EXCLUDE.TRUE)
> 379
>>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',), omit_excluded=nm_db.EXCLUDE.ALL)
> 379
>>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',), omit_excluded=nm_db.EXCLUDE.FLAG)
> 379
Personally I find the description of these flags in notmuch.h not very
clear and don't claim to understand them (this probably explains the
really bad docstring, we should improve those). But I expected to at
least see the EXCLUDE.TRUE one, the default, to work.
Looking at the implementation I don't seem much that could have gone
wrong. However I did notice the bindings fail to check the return code
in one call where it probably should, you could try with this patch?
diff --git a/bindings/python-cffi/notmuch2/_database.py b/bindings/python-cffi/notmuch2/_database.py
index 5ab0f20a..5dbfe68e 100644
--- a/bindings/python-cffi/notmuch2/_database.py
+++ b/bindings/python-cffi/notmuch2/_database.py
@@ -579,7 +579,10 @@ class Database(base.NotmuchObject):
for tag in exclude_tags:
if isinstance(tag, str):
tag = str.encode('utf-8')
- capi.lib.notmuch_query_add_tag_exclude(query_p, tag)
+ ret = capi.lib.notmuch_query_add_tag_exclude(query_p, tag)
+ if ret not in [capi.lib.NOTMUCH_STATUS_SUCCESS,
+ capi.lib.NOTMUCH_STATUS_IGNORED]:
+ raise errors.NotmuchError(ret)
return querymod.Query(self, query_p)
def messages(self, query, *,
To experiment you could also raise the exception for
NOTMUCH_STATUS_IGNORED, even though for the bindings to silently swallow
this is probably the best (this is really the bindings being more
limiting than the C lib, if this bothers ppl we could add another
keyword arg to let this raise as well).
While the above patch is probably good, I feel like it's clutching at
straws and unlikely to solve your problem. Could you try to reproduce
this in plain C or maybe even in pretend-plain-c by using the
capi.lib.notmuch_* calls directly?
Cheers,
Floris
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: New Python bindings (notmuch2 module) fail to exclude tags
2020-11-20 18:17 ` Floris Bruynooghe
@ 2020-11-21 21:51 ` Jorge P. de Morais Neto
2020-11-24 22:33 ` Floris Bruynooghe
0 siblings, 1 reply; 6+ messages in thread
From: Jorge P. de Morais Neto @ 2020-11-21 21:51 UTC (permalink / raw)
To: Floris Bruynooghe, notmuch
Hi Floris.
Em [2020-11-20 sex 19:17:56+0100], Floris Bruynooghe escreveu:
> Looking at the implementation I don't seem much that could have gone
> wrong. However I did notice the bindings fail to check the return code
> in one call where it probably should, you could try with this patch?
>
> diff --git a/bindings/python-cffi/notmuch2/_database.py b/bindings/python-cffi/notmuch2/_database.py
> index 5ab0f20a..5dbfe68e 100644
> --- a/bindings/python-cffi/notmuch2/_database.py
> +++ b/bindings/python-cffi/notmuch2/_database.py
> @@ -579,7 +579,10 @@ class Database(base.NotmuchObject):
> for tag in exclude_tags:
> if isinstance(tag, str):
> tag = str.encode('utf-8')
> - capi.lib.notmuch_query_add_tag_exclude(query_p, tag)
> + ret = capi.lib.notmuch_query_add_tag_exclude(query_p, tag)
> + if ret not in [capi.lib.NOTMUCH_STATUS_SUCCESS,
> + capi.lib.NOTMUCH_STATUS_IGNORED]:
> + raise errors.NotmuchError(ret)
> return querymod.Query(self, query_p)
>
> def messages(self, query, *,
>
After applying your patch, the call to nm_db.count_messages() fails:
AttributeError: cffi library 'notmuch2._capi' has no function, constant or global variable named 'NOTMUCH_STATUS_IGNORED'
However, I then found the bug. The patch below fixes it.
$ diff -u notmuch2-orig/_database.py notmuch2/_database.py
--- notmuch2-orig/_database.py 2020-11-21 18:02:17.560240619 -0300
+++ notmuch2/_database.py 2020-11-21 18:43:44.827879141 -0300
@@ -578,7 +578,7 @@
if exclude_tags is not None:
for tag in exclude_tags:
if isinstance(tag, str):
- tag = str.encode('utf-8')
+ tag = tag.encode('utf-8')
capi.lib.notmuch_query_add_tag_exclude(query_p, tag)
return querymod.Query(self, query_p)
However, I think you should *also* add the error checking. The only
reason my patch omits error checking is that I don't know how to define
capi.lib.NOTMUCH_STATUS_IGNORED. After you apply my patch and also add
the error checking, you could send me the combined patch, so I don't
have to wait for the next release to get the desirable error checking.
Regards
--
- <https://jorgemorais.gitlab.io/justice-for-rms/>
- If an email of mine arrives at your spam box, please notify me.
- Please adopt free/libre formats like PDF, ODF, Org, LaTeX, Opus, WebM and 7z.
- Free/libre software for Replicant, LineageOS and Android: https://f-droid.org
- [[https://www.gnu.org/philosophy/free-sw.html][What is free software?]]
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: New Python bindings (notmuch2 module) fail to exclude tags
2020-11-21 21:51 ` Jorge P. de Morais Neto
@ 2020-11-24 22:33 ` Floris Bruynooghe
2020-11-25 11:07 ` David Bremner
0 siblings, 1 reply; 6+ messages in thread
From: Floris Bruynooghe @ 2020-11-24 22:33 UTC (permalink / raw)
To: Jorge P. de Morais Neto, notmuch
Hi All,
On Sat 21 Nov 2020 at 18:51 -0300, Jorge P. de Morais Neto wrote:
> Hi Floris.
>
> Em [2020-11-20 sex 19:17:56+0100], Floris Bruynooghe escreveu:
>
>> Looking at the implementation I don't seem much that could have gone
>> wrong. However I did notice the bindings fail to check the return code
>> in one call where it probably should, you could try with this patch?
>>
>> diff --git a/bindings/python-cffi/notmuch2/_database.py b/bindings/python-cffi/notmuch2/_database.py
>> index 5ab0f20a..5dbfe68e 100644
>> --- a/bindings/python-cffi/notmuch2/_database.py
>> +++ b/bindings/python-cffi/notmuch2/_database.py
>> @@ -579,7 +579,10 @@ class Database(base.NotmuchObject):
>> for tag in exclude_tags:
>> if isinstance(tag, str):
>> tag = str.encode('utf-8')
>> - capi.lib.notmuch_query_add_tag_exclude(query_p, tag)
>> + ret = capi.lib.notmuch_query_add_tag_exclude(query_p, tag)
>> + if ret not in [capi.lib.NOTMUCH_STATUS_SUCCESS,
>> + capi.lib.NOTMUCH_STATUS_IGNORED]:
>> + raise errors.NotmuchError(ret)
>> return querymod.Query(self, query_p)
>>
>> def messages(self, query, *,
>>
> After applying your patch, the call to nm_db.count_messages() fails:
> AttributeError: cffi library 'notmuch2._capi' has no function, constant or global variable named 'NOTMUCH_STATUS_IGNORED'
>
> However, I then found the bug. The patch below fixes it.
>
> $ diff -u notmuch2-orig/_database.py notmuch2/_database.py
> --- notmuch2-orig/_database.py 2020-11-21 18:02:17.560240619 -0300
> +++ notmuch2/_database.py 2020-11-21 18:43:44.827879141 -0300
> @@ -578,7 +578,7 @@
> if exclude_tags is not None:
> for tag in exclude_tags:
> if isinstance(tag, str):
> - tag = str.encode('utf-8')
> + tag = tag.encode('utf-8')
> capi.lib.notmuch_query_add_tag_exclude(query_p, tag)
> return querymod.Query(self, query_p)
>
>
> However, I think you should *also* add the error checking. The only
> reason my patch omits error checking is that I don't know how to define
> capi.lib.NOTMUCH_STATUS_IGNORED.
Yup, that was missing from _build.py so wouldn't have been available.
Before fixing this I tried to write a test, but I don't know if this
behaves correctly:
I have a database with 3 messages organised in 2 threads and the
following tags:
msg1: all
+- msg3: all, spam
msg2: all
I query '*', so all messages with exclude_tags=['spam']. Querying this
with various flags gives me:
NOTMUCH_EXCLUDE_TRUE: 2 messages, what I expect
NOTMUCH_EXCLUDE_ALL: 2 messages, what I expect
NOTMUCH_EXCLUDE_FLAG: 2 messages, I expected 3
NOTMUCH_EXCLUDE_FLASE: 2 messages, I expected 3
Did I misunderstand the docs? Are these results correct?
Cheers,
Floris
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: New Python bindings (notmuch2 module) fail to exclude tags
2020-11-24 22:33 ` Floris Bruynooghe
@ 2020-11-25 11:07 ` David Bremner
0 siblings, 0 replies; 6+ messages in thread
From: David Bremner @ 2020-11-25 11:07 UTC (permalink / raw)
To: Floris Bruynooghe, Jorge P. de Morais Neto, notmuch
Floris Bruynooghe <flub@devork.be> writes:
>
> I have a database with 3 messages organised in 2 threads and the
> following tags:
>
> msg1: all
> +- msg3: all, spam
> msg2: all
>
> I query '*', so all messages with exclude_tags=['spam']. Querying this
> with various flags gives me:
>
> NOTMUCH_EXCLUDE_TRUE: 2 messages, what I expect
> NOTMUCH_EXCLUDE_ALL: 2 messages, what I expect
> NOTMUCH_EXCLUDE_FLAG: 2 messages, I expected 3
> NOTMUCH_EXCLUDE_FLASE: 2 messages, I expected 3
>
> Did I misunderstand the docs? Are these results correct?
It sounds a bit strange. Can you check against the results of "notmuch
search --output=messages", with the corresponding flag? The numbers
would make more sense if searching for threads rather than messages.
d
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-11-25 11:07 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-11-20 15:54 New Python bindings (notmuch2 module) fail to exclude tags Jorge P. de Morais Neto
2020-11-20 18:02 ` David Bremner
2020-11-20 18:17 ` Floris Bruynooghe
2020-11-21 21:51 ` Jorge P. de Morais Neto
2020-11-24 22:33 ` Floris Bruynooghe
2020-11-25 11:07 ` David Bremner
Code repositories for project(s) associated with this public inbox
https://yhetil.org/notmuch.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).