[afew] announcing afew, an universal tagging solution with some fancy features

[afew] announcing afew, an universal tagging solution with some fancy features

[Justus Winter]

[-- Attachment #1: Type: text/plain, Size: 659 bytes --]

Hey everyone :)

I'd like to introduce my initial tagging approach, afew tags:

https://github.com/teythoon/afew

It has some nifty features, citing the README:

* text classification, magic tags aka the mailing list without server
* spam handling (flush all tags, add spam)
* killed thread handling
* tags posts to lists with lists, $list-id
* autoarchives mails sent from you
* catchall -> remove new, add inbox
* can operate on new messages [default], --all messages or on custom
  query results
* has a --dry-run mode for safe testing
* works with both python2.7 and python3.2

Eagerly waiting for your feedback and patches ;)
Justus

[-- Attachment #2: .signature --]
[-- Type: application/octet-stream, Size: 17 bytes --]

love u alot @,@

Re: [afew] announcing afew,

[Kazuo Teramoto]

        an universal tagging solution with some fancy features
From: Kazuo Teramoto <kaz.rag@gmail.com>
In-Reply-To: <E1RamIG-0003si-K3@thinkbox.jade-hamburg.de>
On 2011-12-14T08:42:36, Justus Winter wrote:
>I'd like to introduce my initial tagging approach, afew tags:
>
>https://github.com/teythoon/afew
>

Very good! Thanks for it. I'm using it as my only filtering/tagging
solutions and works great.

I created a Arch Linux PKGBUILD, it can be find at [aur] (about it,
Justus how you define the afew license? I put a 'unknown' in the license
as I can give it a names like BSD or GPL or WTFPL).

>* works with both python2.7 and python3.2
>

Python 3 support need the patch set you send to the list, correct?

Regards,
Kazuo

[aur]: https://aur.archlinux.org/packages.php?ID=54532

Re: [afew] announcing afew,

[Justus Winter]

[-- Attachment #1: Type: text/plain, Size: 943 bytes --]

Quoting Kazuo Teramoto (2011-12-14 12:24:14)
>an universal tagging solution with some fancy features
>From: Kazuo Teramoto <kaz.rag@gmail.com>
>In-Reply-To: <E1RamIG-0003si-K3@thinkbox.jade-hamburg.de>
>On 2011-12-14T08:42:36, Justus Winter wrote:
>>I'd like to introduce my initial tagging approach, afew tags:
>>
>>https://github.com/teythoon/afew
>>
>
>Very good! Thanks for it. I'm using it as my only filtering/tagging
>solutions and works great.
>
>I created a Arch Linux PKGBUILD, it can be find at [aur]

Cool :)

> (about it,
>Justus how you define the afew license? I put a 'unknown' in the license
>as I can give it a names like BSD or GPL or WTFPL).

afew is licensed under a modern BSD license (I used the openbsd
license). It is included in every source file.

>>* works with both python2.7 and python3.2
>
>Python 3 support need the patch set you send to the list, correct?

Yes.

Cheers,
Justus

[-- Attachment #2: .signature --]
[-- Type: application/octet-stream, Size: 17 bytes --]

love u alot @,@

Re: [afew] announcing afew, an universal tagging solution with some fancy features

[Jani Nikula]

On Wed, 14 Dec 2011 11:42:36 +0100, Justus Winter <4winter@informatik.uni-hamburg.de> wrote:
> https://github.com/teythoon/afew
> 
> It has some nifty features, citing the README:

"""
Basic tagging stuff requires no configuration, just run

$ afew --tag --new

To do this automatically you can add the following hook into your
~/.offlineimaprc:

postsynchook = ionice -c 3 chrt --idle 0 /bin/sh -c "notmuch new && afew
--tag --new"
"""

FYI, "notmuch new" now supports hooks you can use for this. Currently
only in the git repo, though.

BR,
Jani.

Re: [afew] announcing afew, an universal tagging solution with some fancy features

[Patrick Totzke]

Hi Justus,

I have just tried your script and have some questions/remarks about it:

* mkdir -p ~/.config/afew ~/.local/share/afew/categories
  As far as I can see, this is not needed if one doesn't use ClassifyingFilter,
  so its OK not to do this directly from the setup.py somehow.
  But it would be nice if a first run uf `afew --learn` would
  create these (or die gracefully instead of raising the afew.DBACL.BackendError).

* if you renamed the README & co to include the suffix ".md" github would nicely
  render them on the project page.

* I need some more info on how the config works: first, the order in which the
  sections are defined specifies the filter-pipeline yes?
  Secondly, what exactly is a "filter object" in the config?
  Specifically, what do these 3 lines do:
  https://github.com/teythoon/afew/blob/master/docs/tag_filters#L39

* this is surely a user error: I cannot use afew for initial tagging as expected:
  I installed as stated in the README,
  trained some of my existing tags and called `afew --update-reference` and `afew --update`.
  The test with `afew --classify -- ..` works nicely.
  Now I tagges all my mails with is:new, and called `afew --tag --new`.
  This returns immediately without output and all my messages are still tagged new.
  Also, -vv does not result in any output..
  ideas?

Looking forward to replacing my sort script with this.
Thanks,
/p

Re: [afew] announcing afew, an universal tagging solution with some fancy features

[Justus Winter]

[-- Attachment #1: Type: text/plain, Size: 2307 bytes --]

Hey Pazz :)

Quoting Patrick Totzke (2011-12-14 17:59:43)
>Hi Justus,
>
>I have just tried your script and have some questions/remarks about it:
>
>* mkdir -p ~/.config/afew ~/.local/share/afew/categories
>  As far as I can see, this is not needed if one doesn't use ClassifyingFilter,
>  so its OK not to do this directly from the setup.py somehow.
>  But it would be nice if a first run uf `afew --learn` would
>  create these (or die gracefully instead of raising the afew.DBACL.BackendError).

Ah yes. Good point...

>* if you renamed the README & co to include the suffix ".md" github would nicely
>  render them on the project page.

Will do that.

>* I need some more info on how the config works: first, the order in which the
>  sections are defined specifies the filter-pipeline yes?

Yes.

>  Secondly, what exactly is a "filter object" in the config?

Well, you can define filter types either in the configuration file or
in pure python code (see afew/filters). To use them, you need to
create instances of these types.

The config file is just another way of defining python classes and
creating objects. See afew/Settings.py for details.

>  Specifically, what do these 3 lines do:
>  https://github.com/teythoon/afew/blob/master/docs/tag_filters#L39

They create two objects (ShitFilter.0 and ShitFilter.1) of type
ShitFilter (defined in line 32-33). ShitFilter inherits from
SpamFilter and adjusts the message field. ShitFilter.1 also customizes
the message.

>* this is surely a user error: I cannot use afew for initial tagging as expected:
>  I installed as stated in the README,
>  trained some of my existing tags and called `afew --update-reference` and `afew --update`.
>  The test with `afew --classify -- ..` works nicely.
>  Now I tagges all my mails with is:new, and called `afew --tag --new`.
>  This returns immediately without output and all my messages are still tagged new.
>  Also, -vv does not result in any output..
>  ideas?

Is your new tag called 'is:new'? Mine is just called 'new'. That could
be a problem since 'new' as tag for new mails is currently hardcoded
at various locations. I thought it was universally accepted, but then
again hardcoding these things is obviously a bad practice...

Cheers,
Justus

[-- Attachment #2: .signature --]
[-- Type: application/octet-stream, Size: 17 bytes --]

love u alot @,@

Re: [afew] announcing afew, an universal tagging solution with some fancy features

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 650 bytes --]

On Mon, 19 Dec 2011 15:13:51 +0100, Justus Winter <4winter@informatik.uni-hamburg.de> wrote:
> Is your new tag called 'is:new'? Mine is just called 'new'

is: is a search prefix, synonymous with tag:

> That could be a problem since 'new' as tag for new mails is currently
> hardcoded at various locations. I thought it was universally accepted,
> but then again hardcoding these things is obviously a bad practice...

Tags for new messages is configurable with the new.tags config setting.
If you're program needs to know that setting, I suggest it retrieve it
From the notmuch config directly ("notmuch config get new.tags").

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [afew] announcing afew, an universal tagging solution with some fancy features

[Patrick Totzke]

Quoting Jameson Graef Rollins (2011-12-19 17:57:00)
>On Mon, 19 Dec 2011 15:13:51 +0100, Justus Winter <4winter@informatik.uni-hamburg.de> wrote:
>> Is your new tag called 'is:new'? Mine is just called 'new'
>
>is: is a search prefix, synonymous with tag:

Apologies, that was misleading:
my new tag is "new", just like yours.
Also, should i not get some output when calling afew with -vv ?

/p

Re: [afew] announcing afew, an universal tagging solution with some fancy features

[Justus Winter]

[-- Attachment #1: Type: text/plain, Size: 814 bytes --]

Hey Jamie :)

Quoting Jameson Graef Rollins (2011-12-19 18:57:00)
>On Mon, 19 Dec 2011 15:13:51 +0100, Justus Winter <4winter@informatik.uni-hamburg.de> wrote:
>> Is your new tag called 'is:new'? Mine is just called 'new'
>
>is: is a search prefix, synonymous with tag:

Ah, thanks for the clarification.

>> That could be a problem since 'new' as tag for new mails is currently
>> hardcoded at various locations. I thought it was universally accepted,
>> but then again hardcoding these things is obviously a bad practice...
>
>Tags for new messages is configurable with the new.tags config setting.
>If you're program needs to know that setting, I suggest it retrieve it
From the notmuch config directly ("notmuch config get new.tags").

That's a nice touch, it's implemented now :)

Justus

[-- Attachment #2: .signature --]
[-- Type: application/octet-stream, Size: 17 bytes --]

love u alot @,@

Re: [afew] announcing afew, an universal tagging solution with some fancy features

[Justus Winter]

[-- Attachment #1: Type: text/plain, Size: 1903 bytes --]

Hey Pazz,

Quoting Patrick Totzke (2011-12-19 19:17:12)
>Also, should i not get some output when calling afew with -vv ?

Yes you should ;). Here's an example run on my box:

teythoon@thinkbox ~/repos/afew (git)-[master] % python3.2 bin/afew -vv --tag --new
INFO:root:Tagging spam messages
DEBUG:root:Executing query '(tag:new)'
INFO:root:Committing changes to 0 messages
INFO:root:Tagging via classification
DEBUG:root:Executing query '(tag:new)'
DEBUG:root:executing ['dbacl', '-c', ..., '-n']
INFO:root:Committing changes to 1 messages
INFO:root:Looking for messages in killed threads that are not yet killed
DEBUG:root:Executing query '((tag:new)) AND (NOT tag:killed)'
DEBUG:root:Executing query 'thread:"0000000000005d6e" AND tag:killed'
DEBUG:root:Executing query 'thread:"0000000000005d6d" AND tag:killed'
INFO:root:Committing changes to 0 messages
INFO:root:Tagging mailing list posts
DEBUG:root:Executing query '((tag:new)) AND (NOT tag:lists)'
INFO:root:Committing changes to 0 messages
INFO:root:No message specified for filter
DEBUG:root:Executing query '((tag:new)) AND (subject:"[Mafia]")'
INFO:root:Committing changes to 0 messages
INFO:root:Archiving all mails sent by myself
DEBUG:root:Executing query '((tag:new)) AND (from:"4winter@informatik.uni-hamburg.de" OR from:"teythoon@jade-hamburg.de")'
INFO:root:Committing changes to 0 messages
INFO:root:Retags all messages not tagged as junk or killed as inbox
DEBUG:root:Executing query '((tag:new)) AND ((tag:new))'
DEBUG:root:Removing tags new from "Fachschaftsratsvernetzung" <fachschaftsrat@asta.uni-hamburg.de> (new unread) (2011-12-21)
DEBUG:root:Adding tags inbox to "Fachschaftsratsvernetzung" <fachschaftsrat@asta.uni-hamburg.de> (new unread) (2011-12-21)
INFO:root:Committing changes to 1 messages

If you don't get any output at all you might try using the strace
hammer again.

Justus

[-- Attachment #2: .signature --]
[-- Type: application/octet-stream, Size: 17 bytes --]

love u alot @,@

Re: [afew] announcing afew, an universal tagging solution with some fancy features

[Patrick Totzke]

Quoting Justus Winter (2011-12-21 08:53:23)
>Hey Pazz,
>
>Quoting Patrick Totzke (2011-12-19 19:17:12)
>>Also, should i not get some output when calling afew with -vv ?
>
>Yes you should ;). Here's an example run on my box:
Alright, I do get logging output with upstream master thanks.

FYI: I found the problem with afew not finding new mails on my box.
I had the following in my ~/.notmuch-config:


        #       tags    A list (separated by ';') of the tags that will be
        #               added to all messages incorporated by "notmuch new".
        #
        [new]
        tags=new;

I suspect you don't split the config value and thus afew ended up with a false tag "new;".
its just cosmetics but nevertheless..

Thanks for afew! I translated my static filter rules and am very much enjoying the automagic list-id tags :)
/p

Re: [afew] announcing afew, an universal tagging solution with some fancy features

[Justus Winter]

Quoting Patrick Totzke (2012-01-20 11:40:41)
>Quoting Justus Winter (2011-12-21 08:53:23)
>>Hey Pazz,
>>
>>Quoting Patrick Totzke (2011-12-19 19:17:12)
>>>Also, should i not get some output when calling afew with -vv ?
>>
>>Yes you should ;). Here's an example run on my box:
>Alright, I do get logging output with upstream master thanks.
>
>FYI: I found the problem with afew not finding new mails on my box.
>I had the following in my ~/.notmuch-config:
>
>
>        #       tags    A list (separated by ';') of the tags that will be
>        #               added to all messages incorporated by "notmuch new".
>        #
>        [new]
>        tags=new;
>
>I suspect you don't split the config value and thus afew ended up with a false tag "new;".
>its just cosmetics but nevertheless..

Hm, actually it does exactly that:

def get_notmuch_new_tags():
    return notmuch_settings.get('new', 'tags').split(';')

Could you post the relevant log messages?

>Thanks for afew! I translated my static filter rules and am very much enjoying the automagic list-id tags :)
>/p

Cool :)

Cheers,
Justus

Re: [afew] announcing afew, an universal tagging solution with some fancy features

[Kazuo Teramoto]

[-- Attachment #1: Type: text/plain, Size: 636 bytes --]

On Fri, Jan 20, 2012 at 12:26:05PM +0100, Justus Winter wrote:
> Quoting Patrick Totzke (2012-01-20 11:40:41)
> >I suspect you don't split the config value and thus afew ended up
> >with a false tag "new;". its just cosmetics but nevertheless..
> 
> Hm, actually it does exactly that:
> 
> def get_notmuch_new_tags():
>     return notmuch_settings.get('new', 'tags').split(';')

The problem, I think, is caused by how python split the string
~~~~~~~~~~
>>> 'new;'.split(';')
['new', '']
~~~~~~~~~~

So afew pass a query with 'tag:""' and it don't match any message. What
about putting a strip(';') before the split()?

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [afew] announcing afew, an universal tagging solution with some fancy features

[Justus Winter]

Quoting Kazuo Teramoto (2012-01-20 22:55:08)
>On Fri, Jan 20, 2012 at 12:26:05PM +0100, Justus Winter wrote:
>> Quoting Patrick Totzke (2012-01-20 11:40:41)
>> >I suspect you don't split the config value and thus afew ended up
>> >with a false tag "new;". its just cosmetics but nevertheless..
>> 
>> Hm, actually it does exactly that:
>> 
>> def get_notmuch_new_tags():
>>     return notmuch_settings.get('new', 'tags').split(';')
>
>The problem, I think, is caused by how python split the string
>~~~~~~~~~~
>>>> 'new;'.split(';')
>['new', '']
>~~~~~~~~~~
>
>So afew pass a query with 'tag:""' and it don't match any message.

You are totally right, fix pushed.

Thanks,
Justus

Re: [afew] announcing afew, an universal tagging solution with some fancy features

[James Vasile]

I'm tring to figure out how to use afew.  It looks interesting, but
gives me surprising results.

I did `afew --learn spam -- tag:spam` and believe this tells afew that
all the messages matching tag:spam should be classified as spam.  But if
that is right, why would `afew -c spam -- tag:spam` then show me a bunch
of messages that are tagged spam by notmuch but not classified spam by
afew?

How do I train afew when I correct it's mistakes?  Is that what `afew
update` is for?  What does "update the reference category" mean?

Thanks much,
James