whitelisting

whitelisting

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 1333 bytes --]

Hi, folks.  In my on-going war with spam [0], the new battle ground is
false positives: I'm losing too much ham to mis-classification.

For my first line of attack, I would like automatically whitelist every
address to which I have ever sent mail.  I realize this is flawed
(spammers frequently pose as me) but it's my best hope at the moment for
recovering false positives (which is more important than a couple of
additional false negatives).

It's fairly easy to find all such addresses, e.g.:

notmuch address --output=recipients from:jrollins...

But I'm having a hard time coming up with an efficient way to tag mail
coming from any of these address (which total ~4k).  The only command
line way to do it that I've come up with is:

<whitelist.txt xargs -l -I{} notmuch tag -spam tag:spam AND from:{}

This works ok, but takes more than 20s to execute, which will slow down
my inbox processing quite a bit.  I could try to write a python script
to iterate over all tag:spam, extract addresses from those messages, and
match against the whitelist, but I doubt that will be any faster.

Does anyone out there have any better suggestions on how to handle this
kind of white listing?  Anyone come up with any more efficient
algorithms?  Thanks in advance for any suggestions.

jamie.

[0] id:87sj49gha7.fsf@servo.finestructure.net

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: whitelisting

[Steven Allen]

[-- Attachment #1: Type: text/plain, Size: 966 bytes --]


Jameson,

> This works ok, but takes more than 20s to execute, which will slow down
> my inbox processing quite a bit.  I could try to write a python script
> to iterate over all tag:spam, extract addresses from those messages, and
> match against the whitelist, but I doubt that will be any faster.

Instead of iterating over all messages in spam, why not just iterate
over *new* messages (`tag:new`) in your pre hook? That is (pseudo code):

    for message in `notmuch search tag:new and tag:spam`:
        for author in message.headers["From"]: 
            author = clean(author) # Extract the *actual* email address (name@domain).
            # There are probably faster ways to check this...
            if `notmuch count tag:sent and to:author` > 0:
                notmuch tag -spam -- message

That should be reasonably fast.

Note: you probably will have to do this in python because extracting the
from addresses otherwise is a bit of a pain.

- Steven

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: whitelisting

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 790 bytes --]

On Mon, Mar 06 2017, Steven Allen <steven@stebalien.com> wrote:
> Instead of iterating over all messages in spam, why not just iterate
> over *new* messages (`tag:new`) in your pre hook? That is (pseudo code):
>
>     for message in `notmuch search tag:new and tag:spam`:
>         for author in message.headers["From"]: 
>             author = clean(author) # Extract the *actual* email address (name@domain).
>             # There are probably faster ways to check this...
>             if `notmuch count tag:sent and to:author` > 0:
>                 notmuch tag -spam -- message
>
> That should be reasonably fast.

Thanks for the suggestion, Steven.  Yes, my intention was to restrict
over just "new", and yes, it is considerably faster.  Thanks for the
tip.

jamie.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: whitelisting

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 1037 bytes --]

On Mon, Mar 06 2017, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> I could try to write a python script to iterate over all tag:spam,
> extract addresses from those messages, and match against the
> whitelist, but I doubt that will be any faster.

So a custom python script that iterates over all tag:new messages and
matches addresses against the white list is actually quite fast, so
hopefully this will be sufficient for my needs:

  query = 'tag:new AND tag:spam'
  me = get_me_addrs()
  whitelist = list(whitelist_iter())
  with notmuch.Database(mode=notmuch.Database.MODE.READ_WRITE) as db:
      query = db.create_query(query)
      for doc in query.search_messages():
          a = match_addr(doc.get_header('From'))
          if a in whitelist:
              db.begin_atomic()
              doc.remove_tag('spam')
              db.end_atomic()

Guess I should have checked that first, so sorry about the noise.  Still
curious if anyone has come up with any other creative solutions to this
issue though...

jamie.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]