Why Gnus is creating .nnmaildir directory in my maildir?

Why Gnus is creating .nnmaildir directory in my maildir?

[Utkarsh Singh]

Hi,

I am trying to configure Gnus on my GNU/Linux system.  I will first
try to give background for my problem.

1. I am using mbsync/isync to fetch mail.
2. msmtp as `sendmail-program'.
3. '~/.local/share/mail/email-id' as maildir.
4. notmuch as mail indexer.

Overview of my maildir:
~/.local/share/mail/utkarsh190601@gmail.com $ ls
INBOX  [Gmail].Drafts     [Gmail].Sent       [Gmail].Spam     [Gmail].Trash
Sent   [Gmail].Important  [Gmail].Sent Mail  [Gmail].Starred

*Problem: Gnus is creating .nnmaildir directory in my maildir.*

Due to this whenever I try to run 'notmuch new' (command to update
indexer database) it gives several lines long warning:
ignoring non-mail file .nnmaildir/cur/..'

I think there is something wrong with my setup because no other email
client creates extra file when reading maildir (neomutt, notmuch-mua).

Snippets from my config:
;; gnus leave my ~
(setq gnus-home-directory (concat user-emacs-directory "gnus/"))
(setq gnus-directory (concat user-emacs-directory "/gnus/news/"))
(setq gnus-dribble-directory gnus-home-directory)

(setq gnus-select-method '(nnnil ""))
(setq gnus-secondary-select-methods
      '((nntp "news.gwene.org")
        (nnmaildir "utkarsh190601@gmail.com"
                    (directory "~/.local/share/mail/utkarsh190601@gmail.com"))))

--
Utkarsh Singh
http://utkarshsingh.xyz

Re: Why Gnus is creating .nnmaildir directory in my maildir?

[Eric Abrahamsen]

Utkarsh Singh <utkarsh190601@gmail.com> writes:

> Hi,
>
> I am trying to configure Gnus on my GNU/Linux system.  I will first
> try to give background for my problem.
>
> 1. I am using mbsync/isync to fetch mail.
> 2. msmtp as `sendmail-program'.
> 3. '~/.local/share/mail/email-id' as maildir.
> 4. notmuch as mail indexer.
>
> Overview of my maildir:
> ~/.local/share/mail/utkarsh190601@gmail.com $ ls
> INBOX  [Gmail].Drafts     [Gmail].Sent       [Gmail].Spam     [Gmail].Trash
> Sent   [Gmail].Important  [Gmail].Sent Mail  [Gmail].Starred
>
> *Problem: Gnus is creating .nnmaildir directory in my maildir.*
>
> Due to this whenever I try to run 'notmuch new' (command to update
> indexer database) it gives several lines long warning:
> ignoring non-mail file .nnmaildir/cur/..'

The nnmaildir backend uses the .nnmaildir directory to store cached
headers and other information about your maildir messages, for faster
retrieval. Many people don't like this, and/or would prefer more
configurability, but at the moment there's no way around it. For now you
can tell notmuch to ignore that path with the "new.ignore" option in
your notmuch config.

HTH,
Eric

Re: Why Gnus is creating .nnmaildir directory in my maildir?

[Utkarsh Singh]

On 2021-04-13, 08:56 -0700, Eric Abrahamsen <eric@ericabrahamsen.net> wrote:

> Utkarsh Singh <utkarsh190601@gmail.com> writes:
>
>> Hi,
>>
>> I am trying to configure Gnus on my GNU/Linux system.  I will first
>> try to give background for my problem.
>>
>> 1. I am using mbsync/isync to fetch mail.
>> 2. msmtp as `sendmail-program'.
>> 3. '~/.local/share/mail/email-id' as maildir.
>> 4. notmuch as mail indexer.
>>
>> Overview of my maildir:
>> ~/.local/share/mail/utkarsh190601@gmail.com $ ls
>> INBOX  [Gmail].Drafts     [Gmail].Sent       [Gmail].Spam     [Gmail].Trash
>> Sent   [Gmail].Important  [Gmail].Sent Mail  [Gmail].Starred
>>
>> *Problem: Gnus is creating .nnmaildir directory in my maildir.*
>>
>> Due to this whenever I try to run 'notmuch new' (command to update
>> indexer database) it gives several lines long warning:
>> ignoring non-mail file .nnmaildir/cur/..'
>
> The nnmaildir backend uses the .nnmaildir directory to store cached
> headers and other information about your maildir messages, for faster
> retrieval. Many people don't like this, and/or would prefer more
> configurability, but at the moment there's no way around it. For now you
> can tell notmuch to ignore that path with the "new.ignore" option in
> your notmuch config.

Thanks! Everything works as expected now.  For the sake of reference
here is the diff of change I made:

diff --git a/notmuch/.config/notmuch-config b/notmuch/.config/notmuch-config
index 90ec965..72e1c22 100644
--- a/notmuch/.config/notmuch-config
+++ b/notmuch/.config/notmuch-config
@@ -5,7 +5,7 @@ name=Utkarsh Singh
primary_email=utkarsh190601@gmail.com
[new]
tags=unread;inbox;
-ignore=.mbsyncstate;.uidvalidity
+ignore=.mbsyncstate;.uidvalidity;.nnmaildir
[search]
exclude_tags=deleted;spam;
[maildir]
-- 
Utkarsh Singh
http://utkarshsingh.xyz

Re: Why Gnus is creating .nnmaildir directory in my maildir?

[Jean Louis]

* Utkarsh Singh <utkarsh190601@gmail.com> [2021-04-13 18:27]:
> Hi,
> 
> I am trying to configure Gnus on my GNU/Linux system.  I will first
> try to give background for my problem.
> 
> 1. I am using mbsync/isync to fetch mail.
> 2. msmtp as `sendmail-program'.
> 3. '~/.local/share/mail/email-id' as maildir.
> 4. notmuch as mail indexer.
> 
> Overview of my maildir:
> ~/.local/share/mail/utkarsh190601@gmail.com $ ls
> INBOX  [Gmail].Drafts     [Gmail].Sent       [Gmail].Spam     [Gmail].Trash
> Sent   [Gmail].Important  [Gmail].Sent Mail  [Gmail].Starred
> 
> *Problem: Gnus is creating .nnmaildir directory in my maildir.*
> 
> Due to this whenever I try to run 'notmuch new' (command to update
> indexer database) it gives several lines long warning:
> ignoring non-mail file .nnmaildir/cur/..'
> 
> I think there is something wrong with my setup because no other email
> client creates extra file when reading maildir (neomutt, notmuch-mua).

You are totally right, that should not be there, and is not a
standard for Maildirs. With 48000+ maildirs on my side Gnus is totally
disturbing the system and cannot ever even start.

Is useless for Maildirs.

So I keep using mutt within Emacs or from outside term.

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

Sign an open letter in support of Richard M. Stallman
https://stallmansupport.org/
https://rms-support-letter.github.io/

Re: Why Gnus is creating .nnmaildir directory in my maildir?

[Jean Louis]

* Eric Abrahamsen <eric@ericabrahamsen.net> [2021-04-13 19:08]:
> The nnmaildir backend uses the .nnmaildir directory to store cached
> headers and other information about your maildir messages, for faster
> retrieval. Many people don't like this, and/or would prefer more
> configurability, but at the moment there's no way around it. For now you
> can tell notmuch to ignore that path with the "new.ignore" option in
> your notmuch config.

That is terribly flawed design that does not work. Maildir messages
are files, it is quite easy to read subjects, headers when directory
is listed within short time.

What Gnus does to Maildir is abuse to user's data hard disk as there
is no warning about it.


-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

Sign an open letter in support of Richard M. Stallman
https://stallmansupport.org/
https://rms-support-letter.github.io/

Re: Why Gnus is creating .nnmaildir directory in my maildir?

[Eric Abrahamsen]

Jean Louis <bugs@gnu.support> writes:

> * Eric Abrahamsen <eric@ericabrahamsen.net> [2021-04-13 19:08]:
>> The nnmaildir backend uses the .nnmaildir directory to store cached
>> headers and other information about your maildir messages, for faster
>> retrieval. Many people don't like this, and/or would prefer more
>> configurability, but at the moment there's no way around it. For now you
>> can tell notmuch to ignore that path with the "new.ignore" option in
>> your notmuch config.
>
> That is terribly flawed design that does not work. Maildir messages
> are files, it is quite easy to read subjects, headers when directory
> is listed within short time.
>
> What Gnus does to Maildir is abuse to user's data hard disk as there
> is no warning about it.

No need to argue with me about it! But I will note that plenty of GUI
mail clients keep their own cache/data in something like a sqlite
database, alongside the actual message store. nnmaildir is unusable for
large maildirs because it tries to build the entire cache the first time
it is run, and that happens in quadratic time. If you start a new
nnmaildir installation from nothing, and add messages at a "normal"
rate, it's not that bad.

Re: Why Gnus is creating .nnmaildir directory in my maildir?

[Jean Louis]

* Eric Abrahamsen <eric@ericabrahamsen.net> [2021-04-14 01:05]:
> But I will note that plenty of GUI mail clients keep their own
> cache/data in something like a sqlite database, alongside the actual
> message store. nnmaildir is unusable for large maildirs because it
> tries to build the entire cache the first time it is run, and that
> happens in quadratic time. If you start a new nnmaildir installation
> from nothing, and add messages at a "normal" rate, it's not that
> bad.

I have tried all mail clients that deal with Maildirs. Just Gnus does
that kind of mess. None of clients in any GNU/Linux distribution is
similar to it.

The only cache that I know that some client may use is the IMAP header
cache, for some 20000 emails in various folders it is now 55 MB.

Principle of Maildirs is that they are files in a directory. It is
questionable if cache helps there really.


-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

Sign an open letter in support of Richard M. Stallman
https://stallmansupport.org/
https://rms-support-letter.github.io/

maildir package - Re: Why Gnus is creating .nnmaildir directory in my maildir?

[Jean Louis]

There is Emacs package for maildirs:
http://github.com/nicferrier/emacs-maildir

It does not have a concept that impeeds reading Maildirs as Gnus
does. Maildir is extensible by software clients.

IMHO this Emacs package would work better in tabulated-list-mode. It
also need some more features, like easy copy, save, remove into other
Maildir folders and similar.


-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

Sign an open letter in support of Richard M. Stallman
https://stallmansupport.org/
https://rms-support-letter.github.io/

Recommended maildir sorting - Re: Why Gnus is creating .nnmaildir directory in my maildir?

[Jean Louis]

Recommended maildir management is as following:

1. Create new maildir for each email address. For example
   ss@example.com would be saved in a Maildir:

   ~/Maildir/ss@example.com/ with its sub-directories tmp, cur and new
   ~/Maildir/other-person@example.com/ would be saved as Maildir

2. Every email received save by using From: field, to the Maildir
   folder corresponding to From: email address.

3. Every email sent, save into same folder, corresponding to To: email
   address

4. Exception are mailing lists that could be stored by one email
   address, but even then, storing it by person as original poster is
   pretty handy to find out what was conversation about.

The above is most efficient setup I know for long term, like decades
long email management. It allows user to quickly locate all
conversations related to specific email address.

Searching emails and listing them in a specific folder for specific
email address is then way faster compared to storing all emails, let
us say into "Archive" other type of directory trees. But with proper
copy action or symlinking, it is possible to have multiple sorting
methods with same messages when using Maildirs.

IMHO Maildir should be prime and most secure method for management of
emails in Emacs, yet we do not have much software that supports it
natively.

Gnus does not support it natively, it is influencing Maildir folders.

As Maildir is set of 3 directories, it is meant to remain untouched
for purpose of read only accessing it. That is the beauty, just access
directory, list files by reading headers.

For purposes of reply operations, there is nothing to do with the
existing messages, but reading them.

For purposes of copy of the message or moving it to other Maildir,
there is safe method invented by Maildir authors, and that is easy to
do with Emacs.

By following the efficient system above it is then very easy to
connect any kind of address book management to corresponding lists of
emails as they are stored in Maildir folders by their email
address. All conversation related to one person is then visible.

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

Sign an open letter in support of Richard M. Stallman
https://stallmansupport.org/
https://rms-support-letter.github.io/

Re: Why Gnus is creating .nnmaildir directory in my maildir?

[Utkarsh Singh]

On 2021-04-14, 00:01 +0300, Jean Louis <bugs@gnu.support> wrote:

> * Utkarsh Singh <utkarsh190601@gmail.com> [2021-04-13 18:27]:
>> Hi,
>> 
>> I am trying to configure Gnus on my GNU/Linux system.  I will first
>> try to give background for my problem.
>> 
>> 1. I am using mbsync/isync to fetch mail.
>> 2. msmtp as `sendmail-program'.
>> 3. '~/.local/share/mail/email-id' as maildir.
>> 4. notmuch as mail indexer.
>> 
>> Overview of my maildir:
>> ~/.local/share/mail/utkarsh190601@gmail.com $ ls
>> INBOX  [Gmail].Drafts     [Gmail].Sent       [Gmail].Spam     [Gmail].Trash
>> Sent   [Gmail].Important  [Gmail].Sent Mail  [Gmail].Starred
>> 
>> *Problem: Gnus is creating .nnmaildir directory in my maildir.*
>> 
>> Due to this whenever I try to run 'notmuch new' (command to update
>> indexer database) it gives several lines long warning:
>> ignoring non-mail file .nnmaildir/cur/..'
>> 
>> I think there is something wrong with my setup because no other email
>> client creates extra file when reading maildir (neomutt, notmuch-mua).
>
> You are totally right, that should not be there, and is not a
> standard for Maildirs. With 48000+ maildirs on my side Gnus is totally
> disturbing the system and cannot ever even start.
>
> Is useless for Maildirs.
>
> So I keep using mutt within Emacs or from outside term.

If you don't like Gnus and already use Emacs as an editor I would like
to recommend Notmuch Emacs interface(1).  You can also checkout my
configs for notmuch(2).

[1]: https://notmuchmail.org/notmuch-emacs/
[2]: https://github.com/utkarsh181/dotfiles/tree/master/notmuch
-- 
Utkarsh Singh
http://utkarshsingh.xyz

Re: Why Gnus is creating .nnmaildir directory in my maildir?

[Jean Louis]

* Utkarsh Singh <utkarsh190601@gmail.com> [2021-04-14 07:50]:
> If you don't like Gnus and already use Emacs as an editor I would like
> to recommend Notmuch Emacs interface(1).  You can also checkout my
> configs for notmuch(2).

By contrary I would like Gnus handling maildirs and I would like using
it, but it does not.

I have `mu' tools: https://www.djcbsoftware.nl/code/mu/ and `mu4e' as
`mu' for Emacs is not working well if I choose the first option to
list Maildirs, it never ends trying to list 48000 Maildirs. Because it
uses database, and not file system directly.

My access method to emails is based on the concept of "people" and
"conversations" not search terms, though sometimes, rarely, I will
used indexed database.

For example, if I think of Utkarsh, I press a key, write Uktarsh and I
get 4 people, one of them is Singh, some are Dwivedi, Srivastava,
etc. Even if names would be same, people are sorted into groups, I can
see on in "Emacs Users" group. I press key on that one, and all emails
are opened instantly by using mutt.

They can be opened inside emacs as well by using mutt inside of
Terminal, they can be also opened by using Nic Ferrier's Maildir
package.

It is a directory, it should be possible to list it blazing fast
without problems. Gnus sadly does not do that. I am explaining this as
a concept for people to understand and maybe to implement their own
concepts.

Then I can see all previous conversation or interactions between me
and you in March and April.

Thus any searching by To: or From: fields is redundant, as all what is
needed is to open Maildir/specific-person@example.com/ to read those
emails related to To: and From: fields as all will be stored there
inside after replying and reading them first time.

As first time reply and reading is when such emails are in main
~/Maildir/ folder. Then they get moved into their corresponding
folder.

Maybe this system of Maildir management concept by sorting messages
into Maildirs corresponding to the one email address may help you in a
long run. I use it for quite long time, maybe almost 2 decades.

> [1]: https://notmuchmail.org/notmuch-emacs/
> [2]: https://github.com/utkarsh181/dotfiles/tree/master/notmuch

Thank you I will try again. First time I have tried notmuch it did not
work for me. It could not index, and it was back then something
related to 32/64 bits. Now I will try again as I wish to see which
software works better.

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

Sign an open letter in support of Richard M. Stallman
https://stallmansupport.org/
https://rms-support-letter.github.io/

Re: Why Gnus is creating .nnmaildir directory in my maildir?

[Eric Abrahamsen]

Jean Louis <bugs@gnu.support> writes:

> * Eric Abrahamsen <eric@ericabrahamsen.net> [2021-04-14 01:05]:
>> But I will note that plenty of GUI mail clients keep their own
>> cache/data in something like a sqlite database, alongside the actual
>> message store. nnmaildir is unusable for large maildirs because it
>> tries to build the entire cache the first time it is run, and that
>> happens in quadratic time. If you start a new nnmaildir installation
>> from nothing, and add messages at a "normal" rate, it's not that
>> bad.
>
> I have tried all mail clients that deal with Maildirs. Just Gnus does
> that kind of mess. None of clients in any GNU/Linux distribution is
> similar to it.
>
> The only cache that I know that some client may use is the IMAP header
> cache, for some 20000 emails in various folders it is now 55 MB.

The header cache is exactly what the .nnmaildir directory
contains, and that's what takes up the space. It does keep some other
information in there, but I don't think that eats up a lot of disk.