unofficial mirror of notmuch@notmuchmail.org
 help / color / mirror / Atom feed
* New Python bindings (notmuch2 module) fail to exclude tags
@ 2020-11-20 15:54 Jorge P. de Morais Neto
  2020-11-20 18:02 ` David Bremner
  2020-11-20 18:17 ` Floris Bruynooghe
  0 siblings, 2 replies; 6+ messages in thread
From: Jorge P. de Morais Neto @ 2020-11-20 15:54 UTC (permalink / raw)
  To: notmuch

Hi.  I am trying to migrate my Python3 script to the new Python bindings
(notmuch2 module).  However, I cannot obtain a count of messages
matching a query excluding messages that have an exclude tag.  From the
command line:
    $ notmuch count 'is:.bf_spam'
    0

The CLI command correctly counts zero messages having '.bf_spam' tag,
because all such messages also have the excluded 'spam' tag.  But from
Python:

$ python3
Python 3.7.3 (default, Jul 25 2020, 13:03:44) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import notmuch2
>>> nm_db=notmuch2.Database()
>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',))
379
>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',), omit_excluded=nm_db.EXCLUDE.FALSE)
379
>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',), omit_excluded=nm_db.EXCLUDE.TRUE)
379
>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',), omit_excluded=nm_db.EXCLUDE.ALL)
379
>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',), omit_excluded=nm_db.EXCLUDE.FLAG)
379

Am I doing something wrong?

Regards

-- 
- <https://jorgemorais.gitlab.io/justice-for-rms/>
- If an email of mine arrives at your spam box, please notify me.
- Please adopt free/libre formats like PDF, ODF, Org, LaTeX, Opus, WebM and 7z.
- Free/libre software for Replicant, LineageOS and Android: https://f-droid.org
- [[https://www.gnu.org/philosophy/free-sw.html][What is free software?]]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: New Python bindings (notmuch2 module) fail to exclude tags
  2020-11-20 15:54 New Python bindings (notmuch2 module) fail to exclude tags Jorge P. de Morais Neto
@ 2020-11-20 18:02 ` David Bremner
  2020-11-20 18:17 ` Floris Bruynooghe
  1 sibling, 0 replies; 6+ messages in thread
From: David Bremner @ 2020-11-20 18:02 UTC (permalink / raw)
  To: Jorge P. de Morais Neto, notmuch; +Cc: Floris Bruynooghe

"Jorge P. de Morais Neto" <jorge+list@disroot.org> writes:

> Hi.  I am trying to migrate my Python3 script to the new Python bindings
> (notmuch2 module).  However, I cannot obtain a count of messages
> matching a query excluding messages that have an exclude tag.  From the
> command line:
>     $ notmuch count 'is:.bf_spam'
>     0
>
> The CLI command correctly counts zero messages having '.bf_spam' tag,
> because all such messages also have the excluded 'spam' tag.  But from
> Python:
>
> $ python3
> Python 3.7.3 (default, Jul 25 2020, 13:03:44) 
> [GCC 8.3.0] on linux
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import notmuch2
>>>> nm_db=notmuch2.Database()
>>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',))
> 379
>>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',), omit_excluded=nm_db.EXCLUDE.FALSE)
> 379
>>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',), omit_excluded=nm_db.EXCLUDE.TRUE)
> 379
>>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',), omit_excluded=nm_db.EXCLUDE.ALL)
> 379
>>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',), omit_excluded=nm_db.EXCLUDE.FLAG)
> 379
>

It looks like a bug, but I'm not very expert with the python bindings,
so I've CCed the author of the new bindings. It would be great if you
could make a small test case that does not rely on your mail store.

d

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: New Python bindings (notmuch2 module) fail to exclude tags
  2020-11-20 15:54 New Python bindings (notmuch2 module) fail to exclude tags Jorge P. de Morais Neto
  2020-11-20 18:02 ` David Bremner
@ 2020-11-20 18:17 ` Floris Bruynooghe
  2020-11-21 21:51   ` Jorge P. de Morais Neto
  1 sibling, 1 reply; 6+ messages in thread
From: Floris Bruynooghe @ 2020-11-20 18:17 UTC (permalink / raw)
  To: Jorge P. de Morais Neto, notmuch

Hi Jorge,

On Fri 20 Nov 2020 at 12:54 -0300, Jorge P. de Morais Neto wrote:

> Hi.  I am trying to migrate my Python3 script to the new Python bindings
> (notmuch2 module).  However, I cannot obtain a count of messages
> matching a query excluding messages that have an exclude tag.  From the
> command line:
>     $ notmuch count 'is:.bf_spam'
>     0
>
> The CLI command correctly counts zero messages having '.bf_spam' tag,
> because all such messages also have the excluded 'spam' tag.  But from
> Python:
>
> $ python3
> Python 3.7.3 (default, Jul 25 2020, 13:03:44) 
> [GCC 8.3.0] on linux
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import notmuch2
>>>> nm_db=notmuch2.Database()
>>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',))
> 379
>>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',), omit_excluded=nm_db.EXCLUDE.FALSE)
> 379
>>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',), omit_excluded=nm_db.EXCLUDE.TRUE)
> 379
>>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',), omit_excluded=nm_db.EXCLUDE.ALL)
> 379
>>>> nm_db.count_messages('is:.bf_spam', exclude_tags=('spam',), omit_excluded=nm_db.EXCLUDE.FLAG)
> 379

Personally I find the description of these flags in notmuch.h not very
clear and don't claim to understand them (this probably explains the
really bad docstring, we should improve those).  But I expected to at
least see the EXCLUDE.TRUE one, the default, to work.

Looking at the implementation I don't seem much that could have gone
wrong.  However I did notice the bindings fail to check the return code
in one call where it probably should, you could try with this patch?

diff --git a/bindings/python-cffi/notmuch2/_database.py b/bindings/python-cffi/notmuch2/_database.py
index 5ab0f20a..5dbfe68e 100644
--- a/bindings/python-cffi/notmuch2/_database.py
+++ b/bindings/python-cffi/notmuch2/_database.py
@@ -579,7 +579,10 @@ class Database(base.NotmuchObject):
             for tag in exclude_tags:
                 if isinstance(tag, str):
                     tag = str.encode('utf-8')
-                capi.lib.notmuch_query_add_tag_exclude(query_p, tag)
+                ret = capi.lib.notmuch_query_add_tag_exclude(query_p, tag)
+                if ret not in [capi.lib.NOTMUCH_STATUS_SUCCESS,
+                               capi.lib.NOTMUCH_STATUS_IGNORED]:
+                    raise errors.NotmuchError(ret)
         return querymod.Query(self, query_p)
 
     def messages(self, query, *,

To experiment you could also raise the exception for
NOTMUCH_STATUS_IGNORED, even though for the bindings to silently swallow
this is probably the best (this is really the bindings being more
limiting than the C lib, if this bothers ppl we could add another
keyword arg to let this raise as well).

While the above patch is probably good, I feel like it's clutching at
straws and unlikely to solve your problem.  Could you try to reproduce
this in plain C or maybe even in pretend-plain-c by using the
capi.lib.notmuch_* calls directly?


Cheers,
Floris

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: New Python bindings (notmuch2 module) fail to exclude tags
  2020-11-20 18:17 ` Floris Bruynooghe
@ 2020-11-21 21:51   ` Jorge P. de Morais Neto
  2020-11-24 22:33     ` Floris Bruynooghe
  0 siblings, 1 reply; 6+ messages in thread
From: Jorge P. de Morais Neto @ 2020-11-21 21:51 UTC (permalink / raw)
  To: Floris Bruynooghe, notmuch

Hi Floris.

Em [2020-11-20 sex 19:17:56+0100], Floris Bruynooghe escreveu:

> Looking at the implementation I don't seem much that could have gone
> wrong.  However I did notice the bindings fail to check the return code
> in one call where it probably should, you could try with this patch?
>
> diff --git a/bindings/python-cffi/notmuch2/_database.py b/bindings/python-cffi/notmuch2/_database.py
> index 5ab0f20a..5dbfe68e 100644
> --- a/bindings/python-cffi/notmuch2/_database.py
> +++ b/bindings/python-cffi/notmuch2/_database.py
> @@ -579,7 +579,10 @@ class Database(base.NotmuchObject):
>              for tag in exclude_tags:
>                  if isinstance(tag, str):
>                      tag = str.encode('utf-8')
> -                capi.lib.notmuch_query_add_tag_exclude(query_p, tag)
> +                ret = capi.lib.notmuch_query_add_tag_exclude(query_p, tag)
> +                if ret not in [capi.lib.NOTMUCH_STATUS_SUCCESS,
> +                               capi.lib.NOTMUCH_STATUS_IGNORED]:
> +                    raise errors.NotmuchError(ret)
>          return querymod.Query(self, query_p)
>  
>      def messages(self, query, *,
>
After applying your patch, the call to nm_db.count_messages() fails:
    AttributeError: cffi library 'notmuch2._capi' has no function, constant or global variable named 'NOTMUCH_STATUS_IGNORED'

However, I then found the bug.  The patch below fixes it.

$ diff -u notmuch2-orig/_database.py notmuch2/_database.py
--- notmuch2-orig/_database.py	2020-11-21 18:02:17.560240619 -0300
+++ notmuch2/_database.py	2020-11-21 18:43:44.827879141 -0300
@@ -578,7 +578,7 @@
         if exclude_tags is not None:
             for tag in exclude_tags:
                 if isinstance(tag, str):
-                    tag = str.encode('utf-8')
+                    tag = tag.encode('utf-8')
                 capi.lib.notmuch_query_add_tag_exclude(query_p, tag)
         return querymod.Query(self, query_p)


However, I think you should *also* add the error checking.  The only
reason my patch omits error checking is that I don't know how to define
capi.lib.NOTMUCH_STATUS_IGNORED.  After you apply my patch and also add
the error checking, you could send me the combined patch, so I don't
have to wait for the next release to get the desirable error checking.

Regards

-- 
- <https://jorgemorais.gitlab.io/justice-for-rms/>
- If an email of mine arrives at your spam box, please notify me.
- Please adopt free/libre formats like PDF, ODF, Org, LaTeX, Opus, WebM and 7z.
- Free/libre software for Replicant, LineageOS and Android: https://f-droid.org
- [[https://www.gnu.org/philosophy/free-sw.html][What is free software?]]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: New Python bindings (notmuch2 module) fail to exclude tags
  2020-11-21 21:51   ` Jorge P. de Morais Neto
@ 2020-11-24 22:33     ` Floris Bruynooghe
  2020-11-25 11:07       ` David Bremner
  0 siblings, 1 reply; 6+ messages in thread
From: Floris Bruynooghe @ 2020-11-24 22:33 UTC (permalink / raw)
  To: Jorge P. de Morais Neto, notmuch

Hi All,

On Sat 21 Nov 2020 at 18:51 -0300, Jorge P. de Morais Neto wrote:

> Hi Floris.
>
> Em [2020-11-20 sex 19:17:56+0100], Floris Bruynooghe escreveu:
>
>> Looking at the implementation I don't seem much that could have gone
>> wrong.  However I did notice the bindings fail to check the return code
>> in one call where it probably should, you could try with this patch?
>>
>> diff --git a/bindings/python-cffi/notmuch2/_database.py b/bindings/python-cffi/notmuch2/_database.py
>> index 5ab0f20a..5dbfe68e 100644
>> --- a/bindings/python-cffi/notmuch2/_database.py
>> +++ b/bindings/python-cffi/notmuch2/_database.py
>> @@ -579,7 +579,10 @@ class Database(base.NotmuchObject):
>>              for tag in exclude_tags:
>>                  if isinstance(tag, str):
>>                      tag = str.encode('utf-8')
>> -                capi.lib.notmuch_query_add_tag_exclude(query_p, tag)
>> +                ret = capi.lib.notmuch_query_add_tag_exclude(query_p, tag)
>> +                if ret not in [capi.lib.NOTMUCH_STATUS_SUCCESS,
>> +                               capi.lib.NOTMUCH_STATUS_IGNORED]:
>> +                    raise errors.NotmuchError(ret)
>>          return querymod.Query(self, query_p)
>>  
>>      def messages(self, query, *,
>>
> After applying your patch, the call to nm_db.count_messages() fails:
>     AttributeError: cffi library 'notmuch2._capi' has no function, constant or global variable named 'NOTMUCH_STATUS_IGNORED'
>
> However, I then found the bug.  The patch below fixes it.
>
> $ diff -u notmuch2-orig/_database.py notmuch2/_database.py
> --- notmuch2-orig/_database.py	2020-11-21 18:02:17.560240619 -0300
> +++ notmuch2/_database.py	2020-11-21 18:43:44.827879141 -0300
> @@ -578,7 +578,7 @@
>          if exclude_tags is not None:
>              for tag in exclude_tags:
>                  if isinstance(tag, str):
> -                    tag = str.encode('utf-8')
> +                    tag = tag.encode('utf-8')
>                  capi.lib.notmuch_query_add_tag_exclude(query_p, tag)
>          return querymod.Query(self, query_p)
>
>
> However, I think you should *also* add the error checking.  The only
> reason my patch omits error checking is that I don't know how to define
> capi.lib.NOTMUCH_STATUS_IGNORED.

Yup, that was missing from _build.py so wouldn't have been available.

Before fixing this I tried to write a test, but I don't know if this
behaves correctly:

I have a database with 3 messages organised in 2 threads and the
following tags:

msg1: all
  +- msg3: all, spam
msg2: all

I query '*', so all messages with exclude_tags=['spam'].  Querying this
with various flags gives me:

NOTMUCH_EXCLUDE_TRUE: 2 messages, what I expect
NOTMUCH_EXCLUDE_ALL: 2 messages, what I expect
NOTMUCH_EXCLUDE_FLAG: 2 messages, I expected 3
NOTMUCH_EXCLUDE_FLASE: 2 messages, I expected 3

Did I misunderstand the docs?  Are these results correct?


Cheers,
Floris

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: New Python bindings (notmuch2 module) fail to exclude tags
  2020-11-24 22:33     ` Floris Bruynooghe
@ 2020-11-25 11:07       ` David Bremner
  0 siblings, 0 replies; 6+ messages in thread
From: David Bremner @ 2020-11-25 11:07 UTC (permalink / raw)
  To: Floris Bruynooghe, Jorge P. de Morais Neto, notmuch

Floris Bruynooghe <flub@devork.be> writes:

>
> I have a database with 3 messages organised in 2 threads and the
> following tags:
>
> msg1: all
>   +- msg3: all, spam
> msg2: all
>
> I query '*', so all messages with exclude_tags=['spam'].  Querying this
> with various flags gives me:
>
> NOTMUCH_EXCLUDE_TRUE: 2 messages, what I expect
> NOTMUCH_EXCLUDE_ALL: 2 messages, what I expect
> NOTMUCH_EXCLUDE_FLAG: 2 messages, I expected 3
> NOTMUCH_EXCLUDE_FLASE: 2 messages, I expected 3
>
> Did I misunderstand the docs?  Are these results correct?

It sounds a bit strange. Can you check against the results of "notmuch
search --output=messages", with the corresponding flag?  The numbers
would make more sense if searching for threads rather than messages.

d

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2020-11-25 11:07 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-11-20 15:54 New Python bindings (notmuch2 module) fail to exclude tags Jorge P. de Morais Neto
2020-11-20 18:02 ` David Bremner
2020-11-20 18:17 ` Floris Bruynooghe
2020-11-21 21:51   ` Jorge P. de Morais Neto
2020-11-24 22:33     ` Floris Bruynooghe
2020-11-25 11:07       ` David Bremner

unofficial mirror of notmuch@notmuchmail.org

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://yhetil.org/notmuch/0 notmuch/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 notmuch notmuch/ https://yhetil.org/notmuch \
		notmuch@notmuchmail.org
	public-inbox-index notmuch

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.yhetil.org/yhetil.mail.notmuch.general
	nntp://news.gmane.io/gmane.mail.notmuch.general


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git