moving mail to another maildir based on tags and renaming files

moving mail to another maildir based on tags and renaming files

[erik colson]

Hi,

I would like to move mails selected by a notmuch query
(i.e. "tag:movethismailtobox-ThisBox") to *ThisBox*
I need this to somewhat still have usable access through my phone IMAP
client.

I am actually using afew already, but as I have 1000+ mailboxes it is
unmaintainable to create the rules manually.

So I am thinking to either modify the afew-configuration file with
another script (=not nice) or either write a script myself.

It wouldn't be very difficult to generate the necessary
  notmuch search --output=files ... | xargs mv ...
  
but, there is a catch: I use isync to synchronize my mails with my IMAP
server and therefor I must rename every message file when moving it.  As
I am not enough aware of maildir and IMAP servers, I am pretty ignorant
to what the new filename should be.

So I wanted to check if somebody else already wrote some
bash/perl/whatever script which handles this ?

thx
-- 
erik colson

Re: moving mail to another maildir based on tags and renaming files

[inwit]

On Mon May 23, 2022 at 11:02 PM CEST, erik colson wrote:
> So I wanted to check if somebody else already wrote some
> bash/perl/whatever script which handles this ?

I did. I replicate folder and tags with this dirty script:
https://paste.sr.ht/~inwit/018642e5c1cae1e57ae449006b6ed5087ae964d0

You need to be careful with renaming files after moving.

If you have questions, do ask!

Re: moving mail to another maildir based on tags and renaming files

[erik colson]

"inwit" <inwit@sindominio.net> writes:

> If you have questions, do ask!

Do I understand correctly that I only have to remove the U=... part from
the filename when moving a message?  That seems so easy I hardly believe
it ;)

best
-- 
erik colson

Re: moving mail to another maildir based on tags and renaming files

[João Pedro]

[-- Attachment #1: Type: text/plain, Size: 3745 bytes --]

On Mon, May 23 2022 23:02, erik colson <eco@ecocode.net> wrote:

> Hi,
>
> I would like to move mails selected by a notmuch query
> (i.e. "tag:movethismailtobox-ThisBox") to *ThisBox*
> I need this to somewhat still have usable access through my phone IMAP
> client.

What I have is tags that represent IMAP folders, namedely "inbox",
"sent", "trash", "spam" and "all" (representing "All Mail" from Gmail).
And I use a similar set of functions as "inwit <inwit@sindominio.net>",
albeit a bit shorter. I'll be attaching the file.

> I am actually using afew already, but as I have 1000+ mailboxes it is
> unmaintainable to create the rules manually.

I never used afew, but the way I do it doesn't require any manual rules
on the =pre-new= side of things, where I do the moving/renaming. I do
have to do some manual tagging, though, but that's manageable mainly
because I only have 6 mailboxes to handle and they all have similar
structure (and I seem to be able to replicate this structure in most
Mailbox mail providers).

> It wouldn't be very difficult to generate the necessary
>   notmuch search --output=files ... | xargs mv ...

That's pretty much the crux of if, and you should be able to come up
with your own way to suit your own needs. All its needed is basically to
remove the UID part of the filename indeed, though I do it with `sed`
instead of just using string expansion. My setup is basically me trying
to translate Andrew Tropin's setup [1]; see both where he defines
`move-out-untagged-messages` and `move-in-tagged-messages`. Notice that
they both have arguments which I haven't respected, and it would be much
easier to implement it in a proper programming language, but I don't
like to rely on having any language installed cause I'd rather have my
setup work on any Unix-like computer I have to deal with in different
contexts, in which some of them I have a restriction on the installation
of packages). There he is using Guile on Guix Home to generate
configuration files, sort of doing what you alluded as being your first
option, though not really scripting, and with that he has the power of a
general purpose programming language which so happens to also be a Lisp.

> but, there is a catch: I use isync to synchronize my mails with my IMAP
> server and therefor I must rename every message file when moving it.  As
> I am not enough aware of maildir and IMAP servers, I am pretty ignorant
> to what the new filename should be.

I also use isync, and I seem to have been able to keep my phone's mail
structure (looking through a mail app) largely in sync with my personal
files. All I haven't been able to do, mostly because I haven't finished
my mail server setup yet (also doesn't apply to my phone), is to also
sync the notmuch tags, since only a handful of tags are responsible for
moving away my mail.

Also, one other caveat is that you'd best have tagging shortcuts that
remove all other tags that map to a folder, since that may cause
confusion and continuous moving of files, e.g. if you have +inbox +sent
in your mail, on one sync it might be moved to the inbox/ folder and on
the other to the sent/ folder, depending on the order of the tags on
your script. Keep in mind that this only applies to my case, in which I
want to keep 5 different folders in sync. If you just want to move from
inbox/ to somewhere else, especially if you're only concerned with
trash/, junk/ or whatever, it should be simpler, needing only to search
for mails with such tags. I'll also be attaching my tagging
configuration for notmuch in Emacs.

[1] https://github.com/abcdw/rde/blob/master/rde/features/mail.scm#L547

I have a file for the moving of mail portion, given that I use it in
both =pre-new= and =post-new= hooks:


[-- Attachment #2: move-mail script --]
[-- Type: text/plain, Size: 1156 bytes --]

#!/bin/sh
#
# moving mail rules for notmuch
#
# the moving of mails is adapted from:
# https://github.com/abcdw/rde/blob/master/rde/features/mail.scm
alias nmsearch="$(command -v notmuch) search --exclude=false --output=files -- not folder:queued and"

# move out untagged messages
move_out() {
    for tag in inbox sent trash spam; do
        # if the $tag was removed, but the file still is in the folder, move it out
        for f in $(nmsearch path:"/.*\/${tag}\//" and not tag:"$tag" | grep "/${tag}/"); do
            new=$(echo "$f" | sed "s;/${tag}/;/all/;" | sed 's/,U=[0-9]*:/:/')
            [ -f "$f" ] && mv -v "$f" "$new"
        done
    done
}

# move in tagged messages
move_in() {
    for tag in inbox sent trash spam; do
        # if the $tag was added, and the file is not on the folder, move it in
        for f in $(nmsearch not path:"/.*\/${tag}\//" and tag:"$tag"); do
            new=$(echo "$f" | sed "s;/[[:alnum:]]*/cur/;/${tag}/cur/;" | sed 's/,U=[0-9]*:/:/')
            [ -f "$f" ] && mv -v "$f" "$new"
        done
    done
}

case $1 in
    out) move_out ;;
    in) move_in ;;
    *) move_out
       move_in
       ;;
esac

[-- Attachment #3: Type: text/plain, Size: 33 bytes --]


And this is my =pre-new= hook:


[-- Attachment #4: pre-new hook --]
[-- Type: text/plain, Size: 445 bytes --]

#!/bin/sh
#
# commands/hooks to run before `notmuch new`
#
alias nmsearch="$(command -v notmuch) search --exclude=false --output=files -- not folder:queued and"

# do the moving things
~/.config/notmuch/hooks/move-mail

# remove files tagged with 'rm', a tag i use to actually delete emails
for f in $(nmsearch --output=files -- tag:rm); do [ -f "$f" ] && echo rm -i "$f"; done

mbsync -c "${XDG_CONFIG_HOME:-${HOME}/.config}/isync/mbsyncrc" -a

[-- Attachment #5: Type: text/plain, Size: 2395 bytes --]


The relevant part of my Emacs configuration for notmuch is the
following:

#+begin_src emacs-lisp
(defconst notmuch-mark-deleted-tags '("+trash" "-inbox" "-archived" "-sent" "-spam")
  "List of tags to mark for deletion.")

(defconst notmuch-mark-removed-tags (cons "+rm" notmuch-mark-deleted-tags)
  "List of tags to mark for removal.")

(defconst notmuch-mark-spam-tags '("+spam" "-inbox" "-trash" "-sent" "-archived")
  "List of tags to mark as spam.")

(defconst notmuch-mark-flagged-tags '("+flagged")
  "List of tags to mark as important (flagged).")

(defconst notmuch-mark-shopping-tags '("-inbox" "+shopping")
  "List of tags to mark as shopping.
This is related to emails from stuff I buy online.")

(defconst notmuch-mark-financial-tags '("-inbox" "+finance")
  "List of tags to mark as financial.")

(defconst notmuch-mark-meeting-tags '("-inbox" "+meeting")
  "List of tags to mark as meeting.")

(defconst notmuch-mark-account-tags '("-inbox" "+account")
  "List of tags to mark an e-mail as accounts related.
That is, this e-mail in particular is something link a
confirmation of registering or something similar.")

(setq notmuch-archive-tags '("-inbox" "+archived")
      notmuch-message-replied-tags '("+replied")
      notmuch-message-forwarded-tags '("+forwarded" "+sent" "-inbox" "-trash" "-spam")
      notmuch-show-mark-read-tags '("-unread")
      notmuch-draft-tags '("+draft" "-inbox" "-sent" "-trash" "-spam")
      notmuch-tagging-keys
      `((,(kbd "r") notmuch-show-mark-read-tags "Mark as read")
        (,(kbd "u") ("+unread") "Mark as unread")
        ;; (,(kbd "c") notmuch-mark-complete-tags "Complete and archive")
        (,(kbd "d") notmuch-mark-deleted-tags "Mark for deletion")
        (,(kbd "R") notmuch-mark-removed-tags "Mark for removal")
        (,(kbd "f") notmuch-mark-flagged-tags "Flag as important")
        (,(kbd "s") notmuch-mark-spam-tags "Mark as spam")
        (,(kbd "S") notmuch-mark-shopping-tags "Mark as shopping")
        (,(kbd "F") notmuch-mark-financial-tags "Mark as financial")
        (,(kbd "M") notmuch-mark-meeting-tags "Mark as meeting")
        (,(kbd "A") notmuch-mark-account-tags "Mark as account")))
#+end_src

Best regards (and sorry for the long mail),

-- 
João Pedro de Amorim Paula
IT undergraduate at Universidade Federal do Rio Grande do Norte (UFRN)

[-- Attachment #6: Type: text/plain, Size: 0 bytes --]

Re: moving mail to another maildir based on tags and renaming files

[erik colson]

João Pedro <jpedrodeamorim@gmail.com> writes:

> Best regards (and sorry for the long mail),

Hi João,

Thanks for all the explanations and your config!  I will tend to script,
things in fish but meanwhile your code does help a lot!

best
-- 
erik colson\r

Re: moving mail to another maildir based on tags and renaming files

[inwit]

On Tue May 24, 2022 at 12:01 AM CEST, erik colson wrote:
> Do I understand correctly that I only have to remove the U=... part from
> the filename when moving a message?  That seems so easy I hardly believe
> it ;)

Yeah, that is correct. But it proved trickier than it seemed to me (thus the
safeMove2 attempt). 

I must confess that I've holding this tag/folder scheme for over a year now, and
that I'm more and more inclined to reduce the amount of folders that I keep
(I've recently fused 3 or 4 of my biggest folders into Archive, for example).
I'm trying to reduce my use of portable devices, and the truth is that search
engines offered by webmail are slowly getting better, so I feel less exposed by
not having my mail ranged in folders. I guess the whole tag-based paradigm is
also settling down on my mind, although I still feel that its nowadays a bit odd
to rely on a single local copy instead of a cloud-based solution. For this last
problem, there have been some interesting developments recently that I need to
investigate (precisely, the nmbug evolution).

Anyway, notmuch rocks. :)

Re: moving mail to another maildir based on tags and renaming files

[erik colson]

"inwit" <inwit@sindominio.net> writes:

> Yeah, that is correct. But it proved trickier than it seemed to me (thus the
> safeMove2 attempt). 

Yes, I have seen that

> I must confess that I've holding this tag/folder scheme for over a year now, and
> that I'm more and more inclined to reduce the amount of folders that I keep
> (I've recently fused 3 or 4 of my biggest folders into Archive, for example).
> I'm trying to reduce my use of portable devices, and the truth is that search
> engines offered by webmail are slowly getting better, so I feel less exposed by
> not having my mail ranged in folders. I guess the whole tag-based paradigm is
> also settling down on my mind, although I still feel that its nowadays a bit odd
> to rely on a single local copy instead of a cloud-based solution. For this last
> problem, there have been some interesting developments recently that I need to
> investigate (precisely, the nmbug evolution).

I have 1000+ mailboxes linked to projects.  Every project gets a number
and every 'subject' in the project gets its own subfolder.  Take that
down 20+ years and you get 1000+ folders ;)

With notmuch my idea is to bundle the subfolders in the mainfolder so
that would roughly divide the mailboxes by at least 5.  With notmuch the
subfolders are definitely to be tags.  I want to keep the folder for
every project to at least have some kind of fallback in case the notmuch
database gets corrupt..  That is my most important fear with notmuch ;)

I was thinking if notmuch offers some kind of 'dump' for the database,
so it can eventually be restored ?

> Anyway, notmuch rocks. :)

quite a lot, in fact !
-- 
erik colson

Re: moving mail to another maildir based on tags and renaming files

[David Bremner]

erik colson <eco@ecocode.net> writes:

> With notmuch my idea is to bundle the subfolders in the mainfolder so
> that would roughly divide the mailboxes by at least 5.  With notmuch the
> subfolders are definitely to be tags.  I want to keep the folder for
> every project to at least have some kind of fallback in case the notmuch
> database gets corrupt..  That is my most important fear with notmuch ;)
>
> I was thinking if notmuch offers some kind of 'dump' for the database,
> so it can eventually be restored ?

There is in fact the "notmuch dump" command. I recommend everyone who
invests (human) effort in tagging uses it regularly. The output is much
smaller than the notmuch database.

d

Re: moving mail to another maildir based on tags and renaming files

[erik colson]

David Bremner <david@tethera.net> writes:

> There is in fact the "notmuch dump" command. I recommend everyone who
> invests (human) effort in tagging uses it regularly. The output is much
> smaller than the notmuch database.

awesome! thx
-- 
erik colson