* tell me how to do this right (mail sent to lists) @ 2018-10-09 15:02 Jeff Templon 2018-10-09 16:08 ` Daniel Kahn Gillmor 0 siblings, 1 reply; 12+ messages in thread From: Jeff Templon @ 2018-10-09 15:02 UTC (permalink / raw) To: notmuch Hi, The use case: I send a message to a mailing list of which I am a member. I can see via another imap client (in this case MailMate / OS X) that there are two messages in imap, both with the same messageid. One example right now (printing the Subject header below): in my sent mail folder: [Physics@Veldhoven] Inschrijving/Registration Physics@Veldhoven 2019 deadline 9 November 2019 and in my inbox: [pdp] [Physics@Veldhoven] Inschrijving/Registration Physics@Veldhoven 2019: deadline 9 November 2019 However, in notmuch-emacs, I only see the first one. My guess is that notmuch has first seen the one in "sent", indexed that, and then seeing the second one with the same messageId, skips it. I'd prefer to have both, but if I have to pick one or the other, I'd actually prefer to have the second one, as I'd then have [ pdp ] as the start of the subject header for ALL mails to that list, and as an added bonus, afew (should?) then tag them with the list name making them easily searchable. So how does one do this "right"? Thanks, JT ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: tell me how to do this right (mail sent to lists) 2018-10-09 15:02 tell me how to do this right (mail sent to lists) Jeff Templon @ 2018-10-09 16:08 ` Daniel Kahn Gillmor 2018-10-09 20:53 ` Jeff Templon 0 siblings, 1 reply; 12+ messages in thread From: Daniel Kahn Gillmor @ 2018-10-09 16:08 UTC (permalink / raw) To: Jeff Templon, notmuch Hi Jeff-- On Tue 2018-10-09 17:02:05 +0200, Jeff Templon wrote: > I can see via another imap client (in this case MailMate / OS X) that > there are two messages in imap, both with the same messageid. One > example right now (printing the Subject header below): > > in my sent mail folder: > > [Physics@Veldhoven] Inschrijving/Registration Physics@Veldhoven 2019 deadline 9 November 2019 > > and in my inbox: > > [pdp] [Physics@Veldhoven] Inschrijving/Registration Physics@Veldhoven 2019: deadline 9 November 2019 > > However, in notmuch-emacs, I only see the first one. My guess is that > notmuch has first seen the one in "sent", indexed that, and then seeing > the second one with the same messageId, skips it. notmuch doesn't "skip" anything -- it just knows that they are the same e-mail due to message-id matching. it actually indexes both texts. you can see this with: notmuch search --output=files id:$MESSAGEID > I'd prefer to have both, but if I have to pick one or the other, I'd > actually prefer to have the second one, as I'd then have > > [ pdp ] > > as the start of the subject header for ALL mails to that list, and as an > added bonus, afew (should?) then tag them with the list name making them > easily searchable. I'd recommend doing your tagging based on something other than a substring of the subject line -- wouldn't tagging based on the recipient of the message (To: or Cc:) be sufficient to group your mailing list messages under one heading, regardless of which variant notmuch chooses to prefer? i recognize this doesn't answer your proximal question -- and i'd love to hear if someone more knowledgable has a good answer for it. i'm just offering a workaround for what i think is your ultimate goal. all the best, --dkg ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: tell me how to do this right (mail sent to lists) 2018-10-09 16:08 ` Daniel Kahn Gillmor @ 2018-10-09 20:53 ` Jeff Templon 2018-10-09 22:02 ` Carl Worth 0 siblings, 1 reply; 12+ messages in thread From: Jeff Templon @ 2018-10-09 20:53 UTC (permalink / raw) To: Daniel Kahn Gillmor; +Cc: notmuch Hi Daniel, Thanks for answering! On 9 Oct 2018, at 18:08, Daniel Kahn Gillmor wrote: > notmuch doesn't "skip" anything -- it just knows that they are the > same > e-mail due to message-id matching. it actually indexes both texts. > you > can see this with: > > notmuch search --output=files id:$MESSAGEID > Indeed - with the command line tool in this form, I do see both files. Read on :-) >> I'd prefer to have both, but if I have to pick one or the other, I'd >> actually prefer to have the second one, as I'd then have >> >> [ pdp ] >> >> as the start of the subject header for ALL mails to that list, and as >> an >> added bonus, afew (should?) then tag them with the list name making >> them >> easily searchable. > > I'd recommend doing your tagging based on something other than a > substring of the subject line -- wouldn't tagging based on the > recipient > of the message (To: or Cc:) be sufficient to group your mailing list > messages under one heading, regardless of which variant notmuch > chooses > to prefer? So here’s the deal : I am tagging with afew, it has a special mailing list filter, which does the right thing - it tags the copy that was delivered from the mailing list with the list name. However (and maybe this is just the Emacs interface?) when I view the message in notmuch-emacs, it only shows one of them, and this seems to be (always? at least usually) the copy that is Fcc’d to my sent folder and tagged only with “sent” because it doesn’t have a list-id yet. So I’m still looking for how to do this “right”. I must not be the only one that sends mails to mailing lists? JT ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: tell me how to do this right (mail sent to lists) 2018-10-09 20:53 ` Jeff Templon @ 2018-10-09 22:02 ` Carl Worth 2018-10-10 7:16 ` Jeff Templon 0 siblings, 1 reply; 12+ messages in thread From: Carl Worth @ 2018-10-09 22:02 UTC (permalink / raw) To: Jeff Templon, Daniel Kahn Gillmor; +Cc: notmuch [-- Attachment #1: Type: text/plain, Size: 1498 bytes --] On Tue, Oct 09 2018, Jeff Templon wrote: > So here’s the deal : I am tagging with afew, it has a special mailing > list filter, which does the right thing - it tags the copy that was > delivered from the mailing list with the list name. I think you may be mistaken about the above details. If the two messages have the same message ID, then as far as notmuch is concerned that is just a single message, (that happens to be backed by multiple files). > However (and maybe this is just the Emacs interface?) when I view the > message in notmuch-emacs, it only shows one of them, and this seems to > be (always? at least usually) the copy that is Fcc’d to my sent > folder and tagged only with “sent” because it doesn’t have a list-id > yet. It is true that if a single message is backed by multiple files, the notmuch-emacs interface will only display one of them. But even if you changed which one it chooses (which would require some new code as far as I know) shouldn't change what tags you are seeing. > So I’m still looking for how to do this “right”. I must not be the > only one that sends mails to mailing lists? You're certainly not the only one. Hopefully, what you really want doesn't require the ability to maintain separate tags for these two very-similar messages, (since notmuch cannot do that currently). I guess the question I have is: What has given you the idea that afew is tagging one message but not the other? -Carl [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 832 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: tell me how to do this right (mail sent to lists) 2018-10-09 22:02 ` Carl Worth @ 2018-10-10 7:16 ` Jeff Templon 2018-10-10 19:03 ` David Bremner 0 siblings, 1 reply; 12+ messages in thread From: Jeff Templon @ 2018-10-10 7:16 UTC (permalink / raw) To: Carl Worth, Daniel Kahn Gillmor; +Cc: notmuch Hi, Thanks again for the answers. Carl Worth <cworth@cworth.org> writes: > If the two messages have the same message ID, then as far as notmuch is > concerned that is just a single message, (that happens to be backed by > multiple files). OK. > You're certainly not the only one. Hopefully, what you really want > doesn't require the ability to maintain separate tags for these two > very-similar messages, (since notmuch cannot do that currently). I wasn't clear - I don't need different tags. I do want the list-associated tags, even if it only shows me the "sent copy". > I guess the question I have is: What has given you the idea that afew is > tagging one message but not the other? Well, this: medina:~> notmuch search --output=files id:m25zyb89im.fsf@simeto.nikhef.nl /Users/templon/Maildir/Nikhef/Sent/cur/1539098270.M625625000P91131Q71R9d5148db.dhcp-134-239.nikhef.nl:2,S /Users/templon/Maildir/Nikhef/Unclassified Bulk/cur/1539098271.M415561000P91131Q113R5b534386.dhcp-134-239.nikhef.nl:2, medina:~> notmuch search --output=tags id:m25zyb89im.fsf@simeto.nikhef.nl sent So the one in "Sent" is, well, sent :-) The other one (Unclassified Bulk) is the one from the list. There are no list-associated tags. Hence my conclusion is that notmuch and/or afew encounters first the sent copy (maybe because S comes before U?) and tags it, then, when encountering the second file while processing, says in essence "oh i already have this message ID - skip". JT ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: tell me how to do this right (mail sent to lists) 2018-10-10 7:16 ` Jeff Templon @ 2018-10-10 19:03 ` David Bremner 2018-10-10 21:35 ` Jeff Templon 0 siblings, 1 reply; 12+ messages in thread From: David Bremner @ 2018-10-10 19:03 UTC (permalink / raw) To: Jeff Templon, Carl Worth, Daniel Kahn Gillmor; +Cc: notmuch Jeff Templon <templon@nikhef.nl> writes: > Carl Worth <cworth@cworth.org> writes: > >> I guess the question I have is: What has given you the idea that afew is >> tagging one message but not the other? > > Well, this: > > medina:~> notmuch search --output=files id:m25zyb89im.fsf@simeto.nikhef.nl > /Users/templon/Maildir/Nikhef/Sent/cur/1539098270.M625625000P91131Q71R9d5148db.dhcp-134-239.nikhef.nl:2,S > /Users/templon/Maildir/Nikhef/Unclassified Bulk/cur/1539098271.M415561000P91131Q113R5b534386.dhcp-134-239.nikhef.nl:2, > > medina:~> notmuch search --output=tags id:m25zyb89im.fsf@simeto.nikhef.nl > sent > > So the one in "Sent" is, well, sent :-) The other one (Unclassified > Bulk) is the one from the list. There are no list-associated tags. > Hence my conclusion is that notmuch and/or afew encounters first the > sent copy (maybe because S comes before U?) and tags it, then, when > encountering the second file while processing, says in essence "oh i > already have this message ID - skip". The tag is not associated with the file in Sent, it is associated with the message-id. Searching currently for words in the subject of either copy of the message should work, so you should be able to tag the message(-id) as you like. I can't help with afew, but if you have an example that doesn't work on the command line, I'm interested. Display of such messages (and threads) is still less than ideal as it still essentially choose an arbitrary file with that message-id. d ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: tell me how to do this right (mail sent to lists) 2018-10-10 19:03 ` David Bremner @ 2018-10-10 21:35 ` Jeff Templon 2018-10-11 0:50 ` David Bremner ` (2 more replies) 0 siblings, 3 replies; 12+ messages in thread From: Jeff Templon @ 2018-10-10 21:35 UTC (permalink / raw) To: David Bremner, Carl Worth, Daniel Kahn Gillmor; +Cc: notmuch Hi David, all: David Bremner <david@tethera.net> writes: > The tag is not associated with the file in Sent, it is associated > with the message-id. I guess I didn't make myself clear enough, again. I didn't mean that the tag is associated with the file. What I am guessing is something like this: for message in new_messages: if message.id not in database: process message and determine list of tags appy those tags to the messageID new_messages is the list of files that notmuch had not yet seen. notice the if statement. I am guessing, based on the behavior, that this is how it works. So the implication would be: if the first message file encountered for that messageID is the one in Sent, it does not have a ListID field and hence does not get the list tags applied. The second message file (the one sent via the mailing list) DOES have a ListID field, but notmuch "already knows this messageID" so it does not look at that message and the list tags never get applied. I'm not sure whether it's an afew problem or a notmuch problem ... JT ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: tell me how to do this right (mail sent to lists) 2018-10-10 21:35 ` Jeff Templon @ 2018-10-11 0:50 ` David Bremner 2018-10-11 9:22 ` Martin Jambor 2018-10-12 18:02 ` Carl Worth 2 siblings, 0 replies; 12+ messages in thread From: David Bremner @ 2018-10-11 0:50 UTC (permalink / raw) To: Jeff Templon, Carl Worth, Daniel Kahn Gillmor; +Cc: notmuch Jeff Templon <templon@nikhef.nl> writes: > > if the first message file encountered for that messageID is the one in > Sent, it does not have a ListID field and hence does not get the list > tags applied. The second message file (the one sent via the mailing > list) DOES have a ListID field, but notmuch "already knows this > messageID" so it does not look at that message and the list tags never > get applied. > > I'm not sure whether it's an afew problem or a notmuch problem ... > It sounds like an afew problem to me. notmuch proper doesn't know about the List-ID field. d ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: tell me how to do this right (mail sent to lists) 2018-10-10 21:35 ` Jeff Templon 2018-10-11 0:50 ` David Bremner @ 2018-10-11 9:22 ` Martin Jambor 2018-10-11 12:13 ` Jeff Templon 2018-10-12 18:02 ` Carl Worth 2 siblings, 1 reply; 12+ messages in thread From: Martin Jambor @ 2018-10-11 9:22 UTC (permalink / raw) To: Jeff Templon, David Bremner, Carl Worth, Daniel Kahn Gillmor; +Cc: notmuch Hi, I am a notmuch newbie and do not use afew (but have my own post-new hook script), but... On Wed, Oct 10 2018, Jeff Templon wrote: > Hi David, all: > > David Bremner <david@tethera.net> writes: > >> The tag is not associated with the file in Sent, it is associated >> with the message-id. > > I guess I didn't make myself clear enough, again. I didn't mean that > the tag is associated with the file. What I am guessing is something > like this: > > for message in new_messages: > if message.id not in database: > process message and determine list of tags > appy those tags to the messageID > > new_messages is the list of files that notmuch had not yet seen. ...I don't think the above is entirely accurate. IIUC, what happens is that when notmuch sees a message for the first time during "notmuch new", it simply and only applies the tags specified in the tags entry of [new] section in .notmuch-config file to them - the default I believe is "unread;inbox;new". Then the post-hook is executed, which in your case I presume runs afew somehow, and typically that hook goes over all messages with the tag new, process them in an arbitrary way, sometimes adding further tags according to the contents, and eventually drops the new tag. I guess a potential problem is that notmuch sees the message the first time when you send it and it adds the new tag and post-hook processes the outgoing version. When it arrives back, I believe notmuch does not apply the new tag to it anymore. This means that post-hook does not get to process the incoming version, which might be a problem if you want to see stuff added by a remote party, for example when you want to grep the file and look for Sender header in order to mark it as coming from a particular mailing list. That is the reason I do not attempt this in my post-hook and rely on notmuch "tag:new AND to:mailinglist@address" search which is not really precise but at least it should be probably faster. Perhaps it would be a good idea to split the tags entry in [new] section into two, one for tags added to each previously unknown message-id (default would be unread and inbox) and a new one for each new file (default new)? Let me stress again that I am a newbie and can be completely wrong about anything above, so take it with a grain of salt. Thanks, Martin ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: tell me how to do this right (mail sent to lists) 2018-10-11 9:22 ` Martin Jambor @ 2018-10-11 12:13 ` Jeff Templon 0 siblings, 0 replies; 12+ messages in thread From: Jeff Templon @ 2018-10-11 12:13 UTC (permalink / raw) To: Martin Jambor, David Bremner, Carl Worth, Daniel Kahn Gillmor; +Cc: notmuch Hi Martin, Martin Jambor <mjambor@suse.cz> writes: > Then the post-hook is executed, which in your case I presume runs afew > somehow, and typically that hook goes over all messages with the tag > new, process them in an arbitrary way, sometimes adding further tags > according to the contents, and eventually drops the new tag. > > I guess a potential problem is that notmuch sees the message the first > time when you send it and it adds the new tag and post-hook processes > the outgoing version. When it arrives back, I believe notmuch does not > apply the new tag to it anymore. This means that post-hook does not get > to process the incoming version, which might be a problem if you want to > see stuff added by a remote party, for example when you want to grep the > file and look for Sender header in order to mark it as coming from a > particular mailing list. Thanks for your clarification, this is indeed what I meant when I said that what happens is dependent on the order in which notmuch encounters the message files / copies possessing the same messageID. Rereading the afew documentation (not very clear) it hints at this point: "SentMailFilter" It can be used for example to easily tag posts sent to mailing lists which at this stage don’t have List-Id field. without making it explicit, unfortunately, that if you choose not to use this, messages you send to the mailing list won't be tagged in the same way as those received from others through the mailing list. > Perhaps it would be a good idea to split the tags entry in [new] section > into two, one for tags added to each previously unknown message-id > (default would be unread and inbox) and a new one for each new file > (default new)? If I understand you correctly, you're suggesting that newly-detected files always have their messageIDs tagged as "new" even if that messageID has already been "seen"? Maybe that solves my case here, I guess it would depend on how the message files are fed from notmuch to afew. JT ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: tell me how to do this right (mail sent to lists) 2018-10-10 21:35 ` Jeff Templon 2018-10-11 0:50 ` David Bremner 2018-10-11 9:22 ` Martin Jambor @ 2018-10-12 18:02 ` Carl Worth 2018-10-15 11:02 ` Jeff Templon 2 siblings, 1 reply; 12+ messages in thread From: Carl Worth @ 2018-10-12 18:02 UTC (permalink / raw) To: Jeff Templon, David Bremner, Daniel Kahn Gillmor; +Cc: notmuch [-- Attachment #1: Type: text/plain, Size: 4072 bytes --] On Wed, Oct 10 2018, Jeff Templon wrote: >> The tag is not associated with the file in Sent, it is associated >> with the message-id. > > I guess I didn't make myself clear enough, again. I didn't mean that > the tag is associated with the file. What I am guessing is something > like this: Hi Jeff, Thanks for persevering so that we can all try to understand what's happening. I appreciate the patience on all sides. :-) > for message in new_messages: > if message.id not in database: > process message and determine list of tags > appy those tags to the messageID Well, there are actually a couple of different processing loops that you might be describing with the above. Let me try to walk through things: First, there's a loop where "notmuch new" finds previously-unseen files and indexes the content, adding it to the database: for message_file in new_files: message_id = get_headers_message_id (message_file) (message,is_new) = database_lookup (message_id) index_file (message, message_file) if is_new: add_new_message_tags (message) The above pseudo-code is based on the loop in notmuch-new.c:add_files(), add_file() as well as lib/add-message.cc:notmuch_database_index_file() and more or less trying to use naming consistent with the code. Something to not in the above loop: The database_lookup above, (which is actually _notmuch_message_create_for_message_id), can either create a new message object in the database or return an existing object. But, either way, the content of the message will be indexed. So, the significant feature is that notmuch will always be able to search the content it indexes for any message file, (regardless of the order it was encountered given any duplication). However, as can also be seen in the above loop, tags that are added to new messages, (these are as configured in the "new.tags" entry in ~/.notmuch-config) will only be added if the message is new in this pass of notmuch-new. I'm not looking into the code for afew right now, but I can guess a couple of places the undesired bug could be happening: 1. It could be looping over all messages with the "new" tag. And if your sent message gets tagged "new" in a pass before the mailing-list duplicate is present, then afew will not have access to the mailing-list version when it does its processing. Then, later, when a pass does have the mailing-list duplicate present, it won't be considered a "new" message so would not get picked processed in a loop considering messages tagged "new". 2. It's possible that both message files are present at the time that afew does its processing, but that it only opens one of the files to go looking for the List-Id header, (which it must be doing somewhere---as David mentioned, the List-Id header is not ever indexed by notmuch itself). And in the above discussion, I'm assuming that it's even notmuch-new that's doing the detection of new files. Some people use mail flows that have some external mechanism for processing new incoming mail and then calling "notmuch insert" for each one. In conclusion, you have a few different options to get reliable behavior: One option is to use notmuch-based searches to find the mailing-list mail that is of interest for you. To do this you would want to key off of a header that is indexed by notmuch. For example, you could do something like: notmuch tag +my-list-tag to:my-list-recipient-address Another option is to continue to tag messages by inspecting the file (outside of notmuch) to look for a header like List-Id (like you are apparently doing now). To make this reliably, you would simply want to ensure that that processing happens on every new file that is added. And note that the "new" tag as added by "notmuch new" is not reliable for that. That tag _is_ reliable for learning that a new message ID has become available in the database, but is not reliably for know that a new message file has appeared, (for a message ID that was present previously). Does that help explain things? -Carl [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 832 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: tell me how to do this right (mail sent to lists) 2018-10-12 18:02 ` Carl Worth @ 2018-10-15 11:02 ` Jeff Templon 0 siblings, 0 replies; 12+ messages in thread From: Jeff Templon @ 2018-10-15 11:02 UTC (permalink / raw) To: Carl Worth, David Bremner, Daniel Kahn Gillmor; +Cc: notmuch Hi All, Coming back to this, I think (I need to watch it for a week or so) that I have "solved" the mystery. When working from mail, I noticed that sometimes, a mail that I had sent could not be found as "sent". I had borrowed an elisp function from somewhere that marked "interesting messages as sent", and I think what I inadvertently did here was to bypass all the afew tagging for such messages. When I stopped using this elisp function, the messages seem to be tagged correctly again. It's only been a few days (you know the "let's see if it does it again joke, right?") so like I said, I'll keep an eye on it to see whether it's really "fixed" now. Thanks for all your help and patience in this matter! JT ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2018-10-15 11:02 UTC | newest] Thread overview: 12+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2018-10-09 15:02 tell me how to do this right (mail sent to lists) Jeff Templon 2018-10-09 16:08 ` Daniel Kahn Gillmor 2018-10-09 20:53 ` Jeff Templon 2018-10-09 22:02 ` Carl Worth 2018-10-10 7:16 ` Jeff Templon 2018-10-10 19:03 ` David Bremner 2018-10-10 21:35 ` Jeff Templon 2018-10-11 0:50 ` David Bremner 2018-10-11 9:22 ` Martin Jambor 2018-10-11 12:13 ` Jeff Templon 2018-10-12 18:02 ` Carl Worth 2018-10-15 11:02 ` Jeff Templon
Code repositories for project(s) associated with this public inbox https://yhetil.org/notmuch.git/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).