finding file by size

finding file by size

[Mark Janes]

Is there a way to use notmuch to optimize space in my maildir?  I'd like
to find and delete the dozens of messages that were sent to me years ago
with 50MB attachments.

I didn't see a search-term that returns results by size.

I tried to ask this on #notmuch, but got:
  *** #notmuch: Cannot send to channel

thanks,

Mark

Re: finding file by size

[Ralph Seichter]

* Mark Janes:

> I'd like to find and delete the dozens of messages that were sent to
> me years ago with 50MB attachments.

I'm not sure about using Notmuch itself, but this should work:

  find /path/to/maildir -type f -size +50M | xargs rm

Each message in Maildir is stored in a single file, after all.

-Ralph

Re: finding file by size

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1526 bytes --]

On Tue, Nov 06 2018, Ralph Seichter wrote:
> I'm not sure about using Notmuch itself,

Right. Notmuch doesn't currently index (as far as I'm aware) anything
that would be useful for sorting by size.

> but this should work:
>
>   find /path/to/maildir -type f -size +50M | xargs rm

Hmm... I imagine that Mark would be more interested in viewing these
files to ensure they are what he thinks they are before deleting them.

So, capturing the results of that with a notmuch tag would be
a reasonable thing to do. The only trick there is that I don't see any
existing search term to find a message associated with a particular file
name, (we have "path:" and "folder:" to find messages in a specific
directory, but nothing I see for finding the message corresponding to a
specific file).

So, then we could extract the message-id from each file and do a search
based on that I guess?

Here's a (bash) command I just ran on my mail store of over a million
messages that tagged the 8 messages larger than 50MB. It took about a
minute to run (with a warm cache):

  for msg in $(
               for file in $(find . -type f -size +50M);
               do
                 grep -i ^Message-Id $file | sed -e 's/^.*<\(.*\)>.*/\1/';
               done
             );
  do
    notmuch tag +large id:$msg;
  done

With that, I'm able to go through the list from:

  notmuch search tag:large

to investigate whether these large emails are worth keeping.

So, that's obviously not extremely elegant, but it's at least possible.

-Carl


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: finding file by size

[Mark Janes]

Carl Worth <cworth@cworth.org> writes:

> On Tue, Nov 06 2018, Ralph Seichter wrote:
>> I'm not sure about using Notmuch itself,
>
> Right. Notmuch doesn't currently index (as far as I'm aware) anything
> that would be useful for sorting by size.
>
>> but this should work:
>>
>>   find /path/to/maildir -type f -size +50M | xargs rm
>
> Hmm... I imagine that Mark would be more interested in viewing these
> files to ensure they are what he thinks they are before deleting them.

Yes, exactly this.  It's quite nice to see the messages in question.
Looking at the raw message files is not helpful, as the contents are
usually encoded.

> So, capturing the results of that with a notmuch tag would be
> a reasonable thing to do. The only trick there is that I don't see any
> existing search term to find a message associated with a particular file
> name, (we have "path:" and "folder:" to find messages in a specific
> directory, but nothing I see for finding the message corresponding to a
> specific file).

It would be better to have a separate numeric field in xapian similar to
how the date is handled.  Then the user could search for files of
specific sizes.  A more usable feature would be to sort the currently
selected messages by size in emacs or any other front-end.

In the past I've used Thunderbird to archive old/large messages from my
imap server.  It has a column in the UI that allows you to sort files by
size.  I'm just now realizing that notmuch *only* sorts messages by date
(at least in emacs) -- and I haven't missed any other sort functionality
until now...

> So, then we could extract the message-id from each file and do a search
> based on that I guess?
>
> Here's a (bash) command I just ran on my mail store of over a million
> messages that tagged the 8 messages larger than 50MB. It took about a
> minute to run (with a warm cache):
>
>   for msg in $(
>                for file in $(find . -type f -size +50M);
>                do
>                  grep -i ^Message-Id $file | sed -e 's/^.*<\(.*\)>.*/\1/';
>                done
>              );
>   do
>     notmuch tag +large id:$msg;
>   done
>
> With that, I'm able to go through the list from:
>
>   notmuch search tag:large
>
> to investigate whether these large emails are worth keeping.
>
> So, that's obviously not extremely elegant, but it's at least possible.

Thanks for this, I will use your command for my current needs.  Even if
notmuch supported a more elegant feature, I would still need to somehow
locate and remove the messages from my maildir.  I can see how the
separation between the xapian database and the maildir makes it
impossible to support cleaning up the maildir through notmuch.

Re: finding file by size

[David Bremner]

Mark Janes <mark.a.janes@intel.com> writes:

>
> Thanks for this, I will use your command for my current needs.  Even if
> notmuch supported a more elegant feature, I would still need to somehow
> locate and remove the messages from my maildir.  I can see how the
> separation between the xapian database and the maildir makes it
> impossible to support cleaning up the maildir through notmuch.

There is a patch series in progress to add filesize searching, if
someone wanted to work through the review issues:

        id:20170518222708.30032-1-adi@adirat.com

        https://nmbug.notmuchmail.org/nmweb/search/id%3A20170518222708.30032-1-adi%40adirat.com

d

P.S. #notmuch requires nicks to be registered with services, as an
attempt to combat the amount of active spambots.