tell me how to do this right (mail sent to lists)

tell me how to do this right (mail sent to lists)

[Jeff Templon]

Hi,

The use case: I send a message to a mailing list of which I am a member.

I can see via another imap client (in this case MailMate / OS X) that
there are two messages in imap, both with the same messageid.  One
example right now (printing the Subject header below):

in my sent mail folder:

[Physics@Veldhoven] Inschrijving/Registration Physics@Veldhoven 2019 deadline 9 November 2019

and in my inbox:

[pdp] [Physics@Veldhoven] Inschrijving/Registration Physics@Veldhoven 2019: deadline 9 November 2019


However, in notmuch-emacs, I only see the first one.  My guess is that
notmuch has first seen the one in "sent", indexed that, and then seeing
the second one with the same messageId, skips it.  I'd prefer to have
both, but if I have to pick one or the other, I'd actually prefer to
have the second one, as I'd then have

[ pdp ]

as the start of the subject header for ALL mails to that list, and as an
added bonus, afew (should?) then tag them with the list name making them
easily searchable.

So how does one do this "right"?

Thanks,

JT

Re: tell me how to do this right (mail sent to lists)

[Daniel Kahn Gillmor]

Hi Jeff--

On Tue 2018-10-09 17:02:05 +0200, Jeff Templon wrote:
> I can see via another imap client (in this case MailMate / OS X) that
> there are two messages in imap, both with the same messageid.  One
> example right now (printing the Subject header below):
>
> in my sent mail folder:
>
> [Physics@Veldhoven] Inschrijving/Registration Physics@Veldhoven 2019 deadline 9 November 2019
>
> and in my inbox:
>
> [pdp] [Physics@Veldhoven] Inschrijving/Registration Physics@Veldhoven 2019: deadline 9 November 2019
>
> However, in notmuch-emacs, I only see the first one.  My guess is that
> notmuch has first seen the one in "sent", indexed that, and then seeing
> the second one with the same messageId, skips it.

notmuch doesn't "skip" anything -- it just knows that they are the same
e-mail due to message-id matching.  it actually indexes both texts.  you
can see this with:

    notmuch search --output=files id:$MESSAGEID

> I'd prefer to have both, but if I have to pick one or the other, I'd
> actually prefer to have the second one, as I'd then have
>
> [ pdp ]
>
> as the start of the subject header for ALL mails to that list, and as an
> added bonus, afew (should?) then tag them with the list name making them
> easily searchable.

I'd recommend doing your tagging based on something other than a
substring of the subject line -- wouldn't tagging based on the recipient
of the message (To: or Cc:) be sufficient to group your mailing list
messages under one heading, regardless of which variant notmuch chooses
to prefer?

i recognize this doesn't answer your proximal question -- and i'd love
to hear if someone more knowledgable has a good answer for it.  i'm just
offering a workaround for what i think is your ultimate goal.

all the best,

         --dkg

Re: tell me how to do this right (mail sent to lists)

[Jeff Templon]

Hi Daniel,

Thanks for answering!

On 9 Oct 2018, at 18:08, Daniel Kahn Gillmor wrote:

> notmuch doesn't "skip" anything -- it just knows that they are the 
> same
> e-mail due to message-id matching.  it actually indexes both texts.  
> you
> can see this with:
>
>     notmuch search --output=files id:$MESSAGEID
>

Indeed - with the command line tool in this form, I do see both files.  
Read
on :-)

>> I'd prefer to have both, but if I have to pick one or the other, I'd
>> actually prefer to have the second one, as I'd then have
>>
>> [ pdp ]
>>
>> as the start of the subject header for ALL mails to that list, and as 
>> an
>> added bonus, afew (should?) then tag them with the list name making 
>> them
>> easily searchable.
>
> I'd recommend doing your tagging based on something other than a
> substring of the subject line -- wouldn't tagging based on the 
> recipient
> of the message (To: or Cc:) be sufficient to group your mailing list
> messages under one heading, regardless of which variant notmuch 
> chooses
> to prefer?

So here’s the deal : I am tagging with afew, it has a special mailing 
list filter,
which does the right thing - it tags the copy that was delivered from 
the mailing
list with the list name.  However (and maybe this is just the Emacs 
interface?) when
I view the message in notmuch-emacs, it only shows one of them, and this 
seems to be
(always?  at least usually) the copy that is Fcc’d to my sent folder 
and tagged only
with “sent” because it doesn’t have a list-id yet.

So I’m still looking for how to do this “right”.  I must not be 
the only one that sends
mails to mailing lists?

JT

Re: tell me how to do this right (mail sent to lists)

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1498 bytes --]

On Tue, Oct 09 2018, Jeff Templon wrote:
> So here’s the deal : I am tagging with afew, it has a special mailing
> list filter, which does the right thing - it tags the copy that was
> delivered from the mailing list with the list name.

I think you may be mistaken about the above details.

If the two messages have the same message ID, then as far as notmuch is
concerned that is just a single message, (that happens to be backed by
multiple files).

> However (and maybe this is just the Emacs interface?) when I view the
> message in notmuch-emacs, it only shows one of them, and this seems to
> be (always?  at least usually) the copy that is Fcc’d to my sent
> folder and tagged only with “sent” because it doesn’t have a list-id
> yet.

It is true that if a single message is backed by multiple files, the
notmuch-emacs interface will only display one of them. But even if you
changed which one it chooses (which would require some new code as far
as I know) shouldn't change what tags you are seeing.

> So I’m still looking for how to do this “right”.  I must not be the
> only one that sends mails to mailing lists?

You're certainly not the only one. Hopefully, what you really want
doesn't require the ability to maintain separate tags for these two
very-similar messages, (since notmuch cannot do that currently).

I guess the question I have is: What has given you the idea that afew is
tagging one message but not the other?

-Carl


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: tell me how to do this right (mail sent to lists)

[Jeff Templon]

Hi,

Thanks again for the answers.

Carl Worth <cworth@cworth.org> writes:

> If the two messages have the same message ID, then as far as notmuch is
> concerned that is just a single message, (that happens to be backed by
> multiple files).

OK.

> You're certainly not the only one. Hopefully, what you really want
> doesn't require the ability to maintain separate tags for these two
> very-similar messages, (since notmuch cannot do that currently).

I wasn't clear - I don't need different tags.  I do want the
list-associated tags, even if it only shows me the "sent copy".

> I guess the question I have is: What has given you the idea that afew is
> tagging one message but not the other?

Well, this:

medina:~> notmuch search --output=files id:m25zyb89im.fsf@simeto.nikhef.nl
/Users/templon/Maildir/Nikhef/Sent/cur/1539098270.M625625000P91131Q71R9d5148db.dhcp-134-239.nikhef.nl:2,S
/Users/templon/Maildir/Nikhef/Unclassified Bulk/cur/1539098271.M415561000P91131Q113R5b534386.dhcp-134-239.nikhef.nl:2,

medina:~> notmuch search --output=tags id:m25zyb89im.fsf@simeto.nikhef.nl
sent

So the one in "Sent" is, well, sent :-) The other one (Unclassified
Bulk) is the one from the list.  There are no list-associated tags.
Hence my conclusion is that notmuch and/or afew encounters first the
sent copy (maybe because S comes before U?) and tags it, then, when
encountering the second file while processing, says in essence "oh i
already have this message ID - skip".

JT

Re: tell me how to do this right (mail sent to lists)

[David Bremner]

Jeff Templon <templon@nikhef.nl> writes:


> Carl Worth <cworth@cworth.org> writes:
>
>> I guess the question I have is: What has given you the idea that afew is
>> tagging one message but not the other?
>
> Well, this:
>
> medina:~> notmuch search --output=files id:m25zyb89im.fsf@simeto.nikhef.nl
> /Users/templon/Maildir/Nikhef/Sent/cur/1539098270.M625625000P91131Q71R9d5148db.dhcp-134-239.nikhef.nl:2,S
> /Users/templon/Maildir/Nikhef/Unclassified Bulk/cur/1539098271.M415561000P91131Q113R5b534386.dhcp-134-239.nikhef.nl:2,
>
> medina:~> notmuch search --output=tags id:m25zyb89im.fsf@simeto.nikhef.nl
> sent
>
> So the one in "Sent" is, well, sent :-) The other one (Unclassified
> Bulk) is the one from the list.  There are no list-associated tags.
> Hence my conclusion is that notmuch and/or afew encounters first the
> sent copy (maybe because S comes before U?) and tags it, then, when
> encountering the second file while processing, says in essence "oh i
> already have this message ID - skip".

The tag is not associated with the file in Sent, it is associated
with the message-id.

Searching currently for words in the subject of either copy of the
message should work, so you should be able to tag the message(-id) as
you like. I can't help with afew, but if you have an example that
doesn't work on the command line, I'm interested. Display of such
messages (and threads) is still less than ideal as it still essentially
choose an arbitrary file with that message-id.

d

Re: tell me how to do this right (mail sent to lists)

[Jeff Templon]

Hi David, all:

David Bremner <david@tethera.net> writes:

> The tag is not associated with the file in Sent, it is associated
> with the message-id.

I guess I didn't make myself clear enough, again.  I didn't mean that
the tag is associated with the file.  What I am guessing is something
like this:

for message in new_messages:
   if message.id not in database:
      process message and determine list of tags
      appy those tags to the messageID

new_messages is the list of files that notmuch had not yet seen.

notice the if statement.  I am guessing, based on the behavior, that
this is how it works.  So the implication would be:

if the first message file encountered for that messageID is the one in
Sent, it does not have a ListID field and hence does not get the list
tags applied.  The second message file (the one sent via the mailing
list) DOES have a ListID field, but notmuch "already knows this
messageID" so it does not look at that message and the list tags never
get applied.

I'm not sure whether it's an afew problem or a notmuch problem ...

JT

Re: tell me how to do this right (mail sent to lists)

[David Bremner]

Jeff Templon <templon@nikhef.nl> writes:

>
> if the first message file encountered for that messageID is the one in
> Sent, it does not have a ListID field and hence does not get the list
> tags applied.  The second message file (the one sent via the mailing
> list) DOES have a ListID field, but notmuch "already knows this
> messageID" so it does not look at that message and the list tags never
> get applied.
>
> I'm not sure whether it's an afew problem or a notmuch problem ...
>

It sounds like an afew problem to me. notmuch proper doesn't know about
the List-ID field.

d

Re: tell me how to do this right (mail sent to lists)

[Martin Jambor]

Hi,

I am a notmuch newbie and do not use afew (but have my own post-new hook
script), but...

On Wed, Oct 10 2018, Jeff Templon wrote:
> Hi David, all:
>
> David Bremner <david@tethera.net> writes:
>
>> The tag is not associated with the file in Sent, it is associated
>> with the message-id.
>
> I guess I didn't make myself clear enough, again.  I didn't mean that
> the tag is associated with the file.  What I am guessing is something
> like this:
>
> for message in new_messages:
>    if message.id not in database:
>       process message and determine list of tags
>       appy those tags to the messageID
>
> new_messages is the list of files that notmuch had not yet seen.

...I don't think the above is entirely accurate.  IIUC, what happens is
that when notmuch sees a message for the first time during "notmuch
new", it simply and only applies the tags specified in the tags entry of
[new] section in .notmuch-config file to them - the default I believe is
"unread;inbox;new".

Then the post-hook is executed, which in your case I presume runs afew
somehow, and typically that hook goes over all messages with the tag
new, process them in an arbitrary way, sometimes adding further tags
according to the contents, and eventually drops the new tag.

I guess a potential problem is that notmuch sees the message the first
time when you send it and it adds the new tag and post-hook processes
the outgoing version.  When it arrives back, I believe notmuch does not
apply the new tag to it anymore.  This means that post-hook does not get
to process the incoming version, which might be a problem if you want to
see stuff added by a remote party, for example when you want to grep the
file and look for Sender header in order to mark it as coming from a
particular mailing list.

That is the reason I do not attempt this in my post-hook and rely on
notmuch "tag:new AND to:mailinglist@address" search which is not really
precise but at least it should be probably faster.

Perhaps it would be a good idea to split the tags entry in [new] section
into two, one for tags added to each previously unknown message-id
(default would be unread and inbox) and a new one for each new file
(default new)?

Let me stress again that I am a newbie and can be completely wrong about
anything above, so take it with a grain of salt.

Thanks,

Martin

Re: tell me how to do this right (mail sent to lists)

[Jeff Templon]

Hi Martin,

Martin Jambor <mjambor@suse.cz> writes:

> Then the post-hook is executed, which in your case I presume runs afew
> somehow, and typically that hook goes over all messages with the tag
> new, process them in an arbitrary way, sometimes adding further tags
> according to the contents, and eventually drops the new tag.
>
> I guess a potential problem is that notmuch sees the message the first
> time when you send it and it adds the new tag and post-hook processes
> the outgoing version.  When it arrives back, I believe notmuch does not
> apply the new tag to it anymore.  This means that post-hook does not get
> to process the incoming version, which might be a problem if you want to
> see stuff added by a remote party, for example when you want to grep the
> file and look for Sender header in order to mark it as coming from a
> particular mailing list.

Thanks for your clarification, this is indeed what I meant when I said
that what happens is dependent on the order in which notmuch encounters
the message files / copies possessing the same messageID.

Rereading the afew documentation (not very clear) it hints at this
point:

   "SentMailFilter"

   It can be used for example to easily tag posts sent to mailing lists
   which at this stage don’t have List-Id field.

without making it explicit, unfortunately, that if you choose not to use
this, messages you send to the mailing list won't be tagged in the same
way as those received from others through the mailing list.

> Perhaps it would be a good idea to split the tags entry in [new] section
> into two, one for tags added to each previously unknown message-id
> (default would be unread and inbox) and a new one for each new file
> (default new)?

If I understand you correctly, you're suggesting that newly-detected
files always have their messageIDs tagged as "new" even if that
messageID has already been "seen"?  Maybe that solves my case here, I
guess it would depend on how the message files are fed from notmuch to
afew.

JT

Re: tell me how to do this right (mail sent to lists)

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 4072 bytes --]

On Wed, Oct 10 2018, Jeff Templon wrote:
>> The tag is not associated with the file in Sent, it is associated
>> with the message-id.
>
> I guess I didn't make myself clear enough, again.  I didn't mean that
> the tag is associated with the file.  What I am guessing is something
> like this:

Hi Jeff,

Thanks for persevering so that we can all try to understand what's
happening. I appreciate the patience on all sides. :-)

> for message in new_messages:
>    if message.id not in database:
>       process message and determine list of tags
>       appy those tags to the messageID

Well, there are actually a couple of different processing loops that you
might be describing with the above. Let me try to walk through things:

First, there's a loop where "notmuch new" finds previously-unseen files
and indexes the content, adding it to the database:

for message_file in new_files:
  message_id = get_headers_message_id (message_file)
  (message,is_new) = database_lookup (message_id)
  index_file (message, message_file)
  if is_new:
    add_new_message_tags (message)

The above pseudo-code is based on the loop in notmuch-new.c:add_files(),
add_file() as well as lib/add-message.cc:notmuch_database_index_file()
and more or less trying to use naming consistent with the code.

Something to not in the above loop: The database_lookup above, (which is
actually _notmuch_message_create_for_message_id), can either create a
new message object in the database or return an existing object. But,
either way, the content of the message will be indexed. So, the
significant feature is that notmuch will always be able to search the
content it indexes for any message file, (regardless of the order it was
encountered given any duplication).

However, as can also be seen in the above loop, tags that are added
to new messages, (these are as configured in the "new.tags" entry in
~/.notmuch-config) will only be added if the message is new in this pass
of notmuch-new.

I'm not looking into the code for afew right now, but I can guess a
couple of places the undesired bug could be happening:

1. It could be looping over all messages with the "new" tag. And if your
   sent message gets tagged "new" in a pass before the mailing-list
   duplicate is present, then afew will not have access to the
   mailing-list version when it does its processing. Then, later, when a
   pass does have the mailing-list duplicate present, it won't be
   considered a "new" message so would not get picked processed in a
   loop considering messages tagged "new".

2. It's possible that both message files are present at the time that
   afew does its processing, but that it only opens one of the files to
   go looking for the List-Id header, (which it must be doing
   somewhere---as David mentioned, the List-Id header is not ever
   indexed by notmuch itself).

And in the above discussion, I'm assuming that it's even notmuch-new
that's doing the detection of new files. Some people use mail flows that
have some external mechanism for processing new incoming mail and then
calling "notmuch insert" for each one.

In conclusion, you have a few different options to get reliable
behavior:

One option is to use notmuch-based searches to find the mailing-list
mail that is of interest for you. To do this you would want to key off
of a header that is indexed by notmuch. For example, you could do
something like:

	notmuch tag +my-list-tag to:my-list-recipient-address

Another option is to continue to tag messages by inspecting the file
(outside of notmuch) to look for a header like List-Id (like you are
apparently doing now). To make this reliably, you would simply want to
ensure that that processing happens on every new file that is added. And
note that the "new" tag as added by "notmuch new" is not reliable for
that. That tag _is_ reliable for learning that a new message ID has
become available in the database, but is not reliably for know that a
new message file has appeared, (for a message ID that was present
previously).

Does that help explain things?

-Carl






[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: tell me how to do this right (mail sent to lists)

[Jeff Templon]

Hi All,

Coming back to this, I think (I need to watch it for a week or so) that
I have "solved" the mystery.

When working from mail, I noticed that sometimes, a mail that I had sent
could not be found as "sent".  I had borrowed an elisp function from
somewhere that marked "interesting messages as sent", and I think what I
inadvertently did here was to bypass all the afew tagging for such
messages.

When I stopped using this elisp function, the messages seem to be tagged
correctly again.  It's only been a few days (you know the "let's see if
it does it again joke, right?") so like I said, I'll keep an eye on it
to see whether it's really "fixed" now.

Thanks for all your help and patience in this matter!

JT