Gnus does not see new mail in Maildirs

Gnus does not see new mail in Maildirs

[Alexis Praga]

Hi,

When fetching new mails with 'g', Gnus does not see the new mail in my 
inbox and I have to restart it in order to show them.
New mail arrive in ~/Mail/inbox/new and Gnus moves it automatically to 
~/Mail/inbox/cur.
Also, the 'inbox' group disappears when fetch new mail !
Here's my config :

(setq
         gnus-select-method '(nnmaildir "" (directory "~/Mail/"))
         mail-sources '((maildir :path "~/Mail/inbox/" :subdirs ("cur" 
"new")))
)

(setq gnus-secondary-select-methods nil)
(setq gnus-message-archive-group "nnmaildir+inbox:Sent/cur")
(setq gnus-thread-sort-functions
       '((not gnus-thread-sort-by-date)))



Cheers,


---
Alexis Praga
GPG key : 66F9 4BD4 70EC E5D5 2AA4  9097 0673 7CE2 4671 306F

Re: Gnus does not see new mail in Maildirs

[William G. Gardella]

Alexis Praga <alexispraga@mailoo.org> writes:

> Hi,
>
> When fetching new mails with 'g', Gnus does not see the new mail in my
> inbox and I have to restart it in order to show them.
> New mail arrive in ~/Mail/inbox/new and Gnus moves it automatically to
> ~/Mail/inbox/cur.
> Also, the 'inbox' group disappears when fetch new mail !
> Here's my config :
>
> (setq
>         gnus-select-method '(nnmaildir "" (directory "~/Mail/"))
>         mail-sources '((maildir :path "~/Mail/inbox/" :subdirs ("cur"
> "new")))
> )
>

If your MTA is delivering mail directly to your maildir, you should not
also be using the same Maildir as `mail-sources'.  This will cause Gnus
to process the mail twice, with potentially inconsistent results.

As noted in the Info documentation:

> Do _not_ use the same maildir both in `mail-sources' and as an
> `nnmaildir' group.  The results might happen to be useful, but that
> would be by chance, not by design, and the results might be different
> in the future.

(info "(gnus) Maildir")

If the MTA is delivering mail directly to the maildir at ~/Mail/, you
only need the following:

(setq gnus-select-method
      '(nnmaildir ""
		  (directory "~/Maildir/")
		  (get-new-mail nil)))

`mail-sources' should also be nil.  This variable is only useful when
you want Gnus to "pull" the mail from some location, such as an mbox,
pop server, or Maildir at a remote site or elsewhere on the system; it
should not be used when the mail is "pushed" directly to the very same
location where Gnus is storing it.

> (setq gnus-message-archive-group "nnmaildir+inbox:Sent/cur")

is also a bit odd.  You're telling it to use the nnmaildir server
"inbox", which in your config doesn't exist, and you're also telling it
to use a specific subdir inside the maildir, rather than the maildir
itself.

I would rewrite this part of your config as

(setq gnus-message-archive-method gnus-select-method
      gnus-message-archive-group "Sent")

This is very close to the config I use myself and I love it, so I hope
this advice helps.

Re: Gnus does not see new mail in Maildirs

[Alexis Praga]

Thanks for your explanations, it makes much more sense, especially for
the Sent folder.
By using the configuration below, I can finally have new mail in my mailbox by
hitting "/ N" inside the proper group (Inbox in my case).
Can this be done from the group list ?
When I hit "g", it's the same issue as before, that is, the mail is
moved somewhere else.


(setq gnus-select-method
      '(nnmaildir ""
		  (directory "~/Mail/")
		  (get-new-mail nil)))

; Sent mail
(setq gnus-message-archive-method gnus-select-method)
(setq gnus-message-archive-oroup "Sent")

On Fri, Nov 29, 2013 at 09:09:43PM +0000, William G. Gardella wrote:
> Alexis Praga <alexispraga@mailoo.org> writes:
> 
> > Hi,
> >
> > When fetching new mails with 'g', Gnus does not see the new mail in my
> > inbox and I have to restart it in order to show them.
> > New mail arrive in ~/Mail/inbox/new and Gnus moves it automatically to
> > ~/Mail/inbox/cur.
> > Also, the 'inbox' group disappears when fetch new mail !
> > Here's my config :
> >
> > (setq
> >         gnus-select-method '(nnmaildir "" (directory "~/Mail/"))
> >         mail-sources '((maildir :path "~/Mail/inbox/" :subdirs ("cur"
> > "new")))
> > )
> >
> 
> If your MTA is delivering mail directly to your maildir, you should not
> also be using the same Maildir as `mail-sources'.  This will cause Gnus
> to process the mail twice, with potentially inconsistent results.
> 
> As noted in the Info documentation:
> 
> > Do _not_ use the same maildir both in `mail-sources' and as an
> > `nnmaildir' group.  The results might happen to be useful, but that
> > would be by chance, not by design, and the results might be different
> > in the future.
> 
> (info "(gnus) Maildir")
> 
> If the MTA is delivering mail directly to the maildir at ~/Mail/, you
> only need the following:
> 
> (setq gnus-select-method
>       '(nnmaildir ""
> 		  (directory "~/Maildir/")
> 		  (get-new-mail nil)))
> 
> `mail-sources' should also be nil.  This variable is only useful when
> you want Gnus to "pull" the mail from some location, such as an mbox,
> pop server, or Maildir at a remote site or elsewhere on the system; it
> should not be used when the mail is "pushed" directly to the very same
> location where Gnus is storing it.
> 
> > (setq gnus-message-archive-group "nnmaildir+inbox:Sent/cur")
> 
> is also a bit odd.  You're telling it to use the nnmaildir server
> "inbox", which in your config doesn't exist, and you're also telling it
> to use a specific subdir inside the maildir, rather than the maildir
> itself.
> 
> I would rewrite this part of your config as
> 
> (setq gnus-message-archive-method gnus-select-method
>       gnus-message-archive-group "Sent")
> 
> This is very close to the config I use myself and I love it, so I hope
> this advice helps.
> 
> 

-- 
Alexis Praga, PhD Student (CERFACS)
GPG key : 66F9 4BD4 70EC E5D5 2AA4  9097 0673 7CE2 4671 306F

Re: Gnus does not see new mail in Maildirs

[William G. Gardella]

Alexis Praga <alexispraga@mailoo.org> writes:

> Thanks for your explanations, it makes much more sense, especially for
> the Sent folder.
>
> By using the configuration below, I can finally have new mail in my
> mailbox by hitting "/ N" inside the proper group (Inbox in my case).
> Can this be done from the group list ?  When I hit "g", it's the same
> issue as before, that is, the mail is moved somewhere else.

"g" from the *Group* buffer should Just Work now.  All messages in new/,
and all messages not marked as Seen in cur/, will count towards the group's
Unread count.  If this isn't working yet, it's likely that there is
something else strange going on with your config.

As per the Maildir standard, MTAs/MDAs which are compatible with the
Maildir format deliver to new/, and when flags (for example "Seen") are
appended, the mail is moved from new/ to cur/ with the filename altered
appropriately, e.g. ~/Mail/Inbox/new/[some-maildir-id] becomes
~/Mail/Inbox/cur/[some-maildir-id];S when marked Seen by Gnus.

Re: Gnus does not see new mail in Maildirs

[Alexis Praga]

On Fri, Nov 29, 2013 at 11:26:19PM +0000, William G. Gardella wrote:
> "g" from the *Group* buffer should Just Work now.  All messages in new/,
> and all messages not marked as Seen in cur/, will count towards the group's
> Unread count.  If this isn't working yet, it's likely that there is
> something else strange going on with your config.
> 
> As per the Maildir standard, MTAs/MDAs which are compatible with the
> Maildir format deliver to new/, and when flags (for example "Seen") are
> appended, the mail is moved from new/ to cur/ with the filename altered
> appropriately, e.g. ~/Mail/Inbox/new/[some-maildir-id] becomes
> ~/Mail/Inbox/cur/[some-maildir-id];S when marked Seen by Gnus.

I have been doing some tests by moving old mails in Inbox/new and
testing the different configurations.

At the moment, Mutt (which works correctly) moves the mail from
Inbox/new to Inbox/cur and appends a ';S' to the name as you say.
However, when starting Gnus, new mail is directly moved to Inbox/cur
without the Seen flag. And after reading it, the Seen flag is still
not appended !

Can it be due to some file locking ?
Also, would it help to do some kind of reinitialization of gnus ?
 

-- 
Alexis Praga, PhD Student (CERFACS)
GPG key : 66F9 4BD4 70EC E5D5 2AA4  9097 0673 7CE2 4671 306F

Re: Gnus does not see new mail in Maildirs

[W. Greenhouse]

Alexis Praga <alexispraga@mailoo.org> writes:

> On Fri, Nov 29, 2013 at 11:26:19PM +0000, William G. Gardella wrote:
>> "g" from the *Group* buffer should Just Work now.  All messages in new/,
>> and all messages not marked as Seen in cur/, will count towards the group's
>> Unread count.  If this isn't working yet, it's likely that there is
>> something else strange going on with your config.
>> 
>> As per the Maildir standard, MTAs/MDAs which are compatible with the
>> Maildir format deliver to new/, and when flags (for example "Seen") are
>> appended, the mail is moved from new/ to cur/ with the filename altered
>> appropriately, e.g. ~/Mail/Inbox/new/[some-maildir-id] becomes
>> ~/Mail/Inbox/cur/[some-maildir-id];S when marked Seen by Gnus.
>
> I have been doing some tests by moving old mails in Inbox/new and
> testing the different configurations.
>
> At the moment, Mutt (which works correctly) moves the mail from
> Inbox/new to Inbox/cur and appends a ';S' to the name as you say.
> However, when starting Gnus, new mail is directly moved to Inbox/cur
> without the Seen flag. And after reading it, the Seen flag is still
> not appended !

Yeah, now that I look at it, you're correct; Gnus moves from new/ to
cur/ as part of its initial processing of incoming mail (generating news
overview files from the metadata and stashing them away in .nnmaildir
inside the Maildir).  It is only marked as read, though when quitting or
manually updating the group, with q or x respectively.

> Can it be due to some file locking ?
> Also, would it help to do some kind of reinitialization of gnus ?

Are you keeping the *Summary ...* buffer for your Maildir open during
this testing?  If so, I think that's the problem.  Changes to a
message's unread or flagged state are not actually committed to the
Maildir until you do x (`gnus-summary-limit-to-unread') or q
(`gnus-summary-exit').

Re: Gnus does not see new mail in Maildirs

[Alexis Praga]

First, I finally managed to get Gnus working properly. For that, I
removed configuration files (.newsrc, .newrs.eld) and after restarting,
new mail was properly displayed with gnus-group-get-new-news.

> Are you keeping the *Summary ...* buffer for your Maildir open during
> this testing?  If so, I think that's the problem.  Changes to a
> message's unread or flagged state are not actually committed to the
> Maildir until you do x (`gnus-summary-limit-to-unread') or q
> (`gnus-summary-exit').
No, I restarted Gnus each time. But that explains the difference.

Anyway, thanks, you have been very helpful !


-- 
Alexis Praga, PhD Student (CERFACS)
GPG key : 66F9 4BD4 70EC E5D5 2AA4  9097 0673 7CE2 4671 306F

Re: Gnus does not see new mail in Maildirs

[Emanuel Berg]

Alexis Praga <alexispraga@mailoo.org> writes:

> Thanks for your explanations, it makes much more
> sense, especially for the Sent folder.  By using the
> configuration below, I can finally have new mail in
> my mailbox by hitting "/ N" inside the proper group
> (Inbox in my case).

"/ N" in the summary buffer seems to be
`gnus-summary-insert-new-articles' (check this out with
C-h k and then / N) so no, in the group buffer you
can't do that because there is no argument where you
can provide the group (in your case the inbox).

You could of course set that up based on the cursor
position and so on but there must be a more
uncomplicated way to do what you wish.

> Can this be done from the group list?

Yes, it seems backward, what you are doing. In the
group buffer, what happens if you just hit RET (for
`gnus-group-select-group')?

> When I hit "g", it's the same issue as before, that
> is, the mail is moved somewhere else.

You mean "g" in the group buffer? Can you check out
what command that runs (with the method just
mentioned)? Also, can it be the mail is not *moved*, it
is just not *shown*? I only have new mails in my
mail.misc group. To get the archive "back" (displayed)
I use C (for `gnus-group-catchup-current-all') and then
RET RET with the cursor on that group, in the group
buffer.

-- 
Emanuel Berg, programmer-for-rent. CV, projects, etc at uXu
underground experts united:  http://user.it.uu.se/~embe8573

Re: Gnus does not see new mail in Maildirs

[Emanuel Berg]

William G. Gardella <wgg2@member.fsf.org> writes:

> This is very close to the config I use myself and I
> love it, so I hope this advice helps.

I remember talking to you on gnu.emacs.gnus and I
didn't quite understand you, and that was because your
reasoning applied to nnmaildir, and mine to nnml.

In the Gnus manual FAQ on different solutions for mail
[1], I read:

> nnmaildir is largely similar to nnml, with some
> notable differences. Each message is stored in a
> separate file, but the filename is unrelated to the
> article number in Gnus. nnmaildir also stores the
> equivalent of nnml's overview files in one file per
> article, so it uses about twice as many inodes as
> nnml.

Well, that doesn't seem to be that radical a
difference.

[1] http://gnus.org/manual/gnus_202.html

-- 
Emanuel Berg, programmer-for-rent. CV, projects, etc at uXu
underground experts united:  http://user.it.uu.se/~embe8573

Re: Gnus does not see new mail in Maildirs

[William G. Gardella]

Emanuel Berg <embe8573@student.uu.se> writes:

> William G. Gardella <wgg2@member.fsf.org> writes:
>
>> This is very close to the config I use myself and I
>> love it, so I hope this advice helps.
>
> I remember talking to you on gnu.emacs.gnus and I
> didn't quite understand you, and that was because your
> reasoning applied to nnmaildir, and mine to nnml.
>
> In the Gnus manual FAQ on different solutions for mail
> [1], I read:

[...]

> Well, that doesn't seem to be that radical a
> difference.

The main difference between Maildir and MH-style one-message-per-file
formats like nnml is that Maildir is designed to handle concurrent
manipulations by multiple clients or mail transport/mail delivery agents
at the same time.  Rather than Gnus handling final local mail delivery
to the folders where you're keeping the mail, it's possible for system
daemons to take care of that and have Gnus (and other Maildir-compatible
MUAs, mail indexing utilities, etc.) just sit back and watch. :) Your
system's mail server can be delivering mail directly to a user's Maildir
even as the user's MUA reads, moves, flags, or deletes messages in the
same Maildir.  To read more about the Maildir specification, have a look
at http://cr.yp.to/proto/maildir.html.

In my setup, for example, mail arrives at my system's MTA, sendmail,
either by fetchmail from a remote POP site or by direct delivery to the
MTA.  As the final step in sendmail's processing of the incoming mail,
sendmail invokes procmail using my ~/.procmailrc, which instructs it to
deliver to several Maildirs based on information in the headers.  (Note
that some newer MTAs, e.g. Postfix, often support delivery to Maildirs
directly, so the additional local mail delivery agent like procmail may
be omittted.)

~/.procmailrc is here:

--8<---------------cut here---------------start------------->8---
:0 c
* ^X-Spam-Status: Yes
Maildir/spam/
:0 Ec
* ^List-Unsubscribe:.*
Maildir/lists/
:0 Ec
Maildir/mail/
:0
| notmuch new
--8<---------------cut here---------------end--------------->8---

These rules tell it to copy SpamAssassin-suspected spam to
~/Maildir/spam/, mail with a List-Unsubscribe header to
~/Maildir/lists/, and, as the fallback rule, everything else (hopefully
actual personal correspondence!) to ~/Maildir/mail/.  The final rule
runs the "notmuch" indexing software <http://notmuchmail.org>, which I
use together with Gnus's `nnir' library for fast search, on all the
incoming mail, and purges it from sendmail's spool.

Gnus mail-handling config in further detail is here:

--8<---------------cut here---------------start------------->8---
(setq gnus-select-method
      '(nnmaildir ""
		  (directory "~/Maildir/")
		  (get-new-mail nil)
		  (nnir-search-engine notmuch))
      nnmail-split-methods
      '(("spam" "^X-Spam-Status: Yes")
	("lists" "^List-Unsubscribe:.*")
	("mail" ""))
      gnus-secondary-select-methods
      '((nntp "news.gmane.org"))
      gnus-message-archive-method gnus-select-method
      gnus-message-archive-group "mail"
      nnir-notmuch-remove-prefix "/home/wgg/Maildir/"
      gnus-summary-respool-default-method 'nnmaildir)
--8<---------------cut here---------------end--------------->8---

It seems like needless redundancy, but I reproduce the procmail
splitting rules in Gnus as well so that when "respooling" mail into the
Maildir backend, e.g. from downloaded mbox archives of mailing list
posts, it is automatically filtered into the appropriate folders (even
though it didn't arrive through the sendmail->procmail pipeline).

As a subtle form of new-mail notification in the mode-line, I also use
`display-time-mode' together with an appropriate setting of
`display-time-mail-directory':

(setq display-time-mail-directory "~/Maildir/mail/new/")

`display-time' doesn't directly grok Maildir as far as I know, but the
above is a kludge so that only personal mails (those filtered to
"~/Maildir/mail"), which have not been renamed yet by Gnus or another
Maildir client (and are thus still in new/), trigger the notification.

Re: Gnus does not see new mail in Maildirs

[Emanuel Berg]

"William G. Gardella" <wgg2@member.fsf.org> writes:

> The main difference between Maildir and MH-style
> one-message-per-file formats like nnml is that
> Maildir is designed to handle concurrent
> manipulations by multiple clients or mail
> transport/mail delivery agents at the same time.
> Rather than Gnus handling final local mail delivery
> to the folders where you're keeping the mail, it's
> possible for system daemons to take care of that and
> have Gnus (and other Maildir-compatible MUAs, mail
> indexing utilities, etc.) just sit back and watch. :)

So it is a less interactive system and you use Gnus
strictly as an interface. Perhaps that is something
that is (more) beneficial at the server level and/or if
you manage huge amounts of mails, as in listbots and
the like?

-- 
Emanuel Berg, programmer-for-rent. CV, projects, etc at uXu
underground experts united:  http://user.it.uu.se/~embe8573

Emacs and solarized

[Alexis Praga]

Hi,

Does anyone has a fully fonctional configuration for the solarized theme 
on Emacs 23 ?
I used the github version :
https://github.com/sellout/emacs-color-theme-solarized

It works for the GUI. For a console Emacs, I have to start it with 
'TERM=xterm-256colors emacs -new' to make it work under tmux.
However, I couldn't use it with 'emacs --daemon'.


---
Alexis Praga
GPG key : 66F9 4BD4 70EC E5D5 2AA4  9097 0673 7CE2 4671 306F

Re: Emacs and solarized

[Ian van der Neut]

On Mon, Dec 9, 2013 at 10:13 AM, Alexis Praga <alexispraga@mailoo.org>wrote:

> Hi,
>
> Does anyone has a fully fonctional configuration for the solarized theme
> on Emacs 23 ?
> I used the github version :
> https://github.com/sellout/emacs-color-theme-solarized
>
> It works for the GUI. For a console Emacs, I have to start it with
> 'TERM=xterm-256colors emacs -new' to make it work under tmux.
> However, I couldn't use it with 'emacs --daemon'.
>
>
> Hi Alexis,

the one in sellout's repository doesn't work on Emacs 23 in daemon mode. I
found one that does:

 https://github.com/purcell/color-theme-sanityinc-solarized

However, I've moved on to emacs 24 a while ago, so no idea what the status
is.

Ian.

-- 
One man's "magic" is another man's engineering. "Supernatural" is a null
word.
-- Excerpt from the notebooks of Lazarus Long, from Robert Heinlein's "Time
Enough for Love"

Re: Emacs and solarized

[Alexis Praga]

> Hi Alexis,
> 
> the one in sellout's repository doesn't work on Emacs 23 in daemon 
> mode. I
> found one that does:
> 
>  https://github.com/purcell/color-theme-sanityinc-solarized

With this version, the colorscheme is wrong for both daemon mode and 
"console" mode (outside tmux).
However, it is consistent :)

Cheers,


---
Alexis Praga
GPG key : 66F9 4BD4 70EC E5D5 2AA4  9097 0673 7CE2 4671 306F

Re: Emacs and solarized

[Alexis Praga]

Update : emacs --daemon does not load the solarized theme but 
emacsclient does.
If it doesn't, it only needs to be done once with M-x 
color-theme-solarized-dark.


---
Alexis Praga
GPG key : 66F9 4BD4 70EC E5D5 2AA4  9097 0673 7CE2 4671 306F

Re: Emacs and solarized

[William G. Gardella]

Alexis Praga <alexispraga@mailoo.org> writes:

> Update : emacs --daemon does not load the solarized theme but
> emacsclient does.
> If it doesn't, it only needs to be done once with M-x 
> color-theme-solarized-dark.

Sometimes color themes (of both the old and new varieties) can't fully
load on --daemon, because --daemon's apparent color depth is that of a
standard TTY (8 colors, 16 counting bold).  If the theme attempts to set
things like cursor or mouse color (which cannot be set in a text
console/terminal environment), this will be a particularly obvious breakage.

I started to use something like this:

--8<---------------cut here---------------start------------->8---
(defun wg/setup-themes (frame)
  "Kludge themes, e.g. to get things right on emacsclient."
  (ignore frame)
  (mapcar 'load-theme custom-enabled-themes))


(add-hook 'after-make-frame-functions 'wg/setup-themes)
--8<---------------cut here---------------end--------------->8---

Re: Emacs and solarized

[Alexis Praga]

> Sometimes color themes (of both the old and new varieties) can't fully
> load on --daemon, because --daemon's apparent color depth is that of a
> standard TTY (8 colors, 16 counting bold).  If the theme attempts to 
> set
> things like cursor or mouse color (which cannot be set in a text
> console/terminal environment), this will be a particularly obvious 
> breakage.
> 
> I started to use something like this:
> 
> --8<---------------cut here---------------start------------->8---
> (defun wg/setup-themes (frame)
>   "Kludge themes, e.g. to get things right on emacsclient."
>   (ignore frame)
>   (mapcar 'load-theme custom-enabled-themes))
> 
> 
> (add-hook 'after-make-frame-functions 'wg/setup-themes)
> --8<---------------cut here---------------end--------------->8---

I tried the following :

(defun wg/setup-themes (frame)
     "Kludge themes, e.g. to get things right on emacsclient."
     (ignore frame)
     (color-theme-solarized-dark))

(add-hook 'after-make-frame-functions 'wg/setup-themes)

But to no avail.

Cheers,


---
Alexis Praga
GPG key : 66F9 4BD4 70EC E5D5 2AA4  9097 0673 7CE2 4671 306F

Re: Emacs and solarized

[W. Greenhouse]

> I tried the following :
>
> (defun wg/setup-themes (frame)
>     "Kludge themes, e.g. to get things right on emacsclient."
>     (ignore frame)
>     (color-theme-solarized-dark))
>
> (add-hook 'after-make-frame-functions 'wg/setup-themes)
>
> But to no avail.

Note: only works with emacs24 themes.  You'd need a different
theme-kludging function for color-theme-* themes, but same idea.

Re: Emacs and solarized

[Alexis Praga]

>> (defun wg/setup-themes (frame)
>>     "Kludge themes, e.g. to get things right on emacsclient."
>>     (ignore frame)
>>     (color-theme-solarized-dark))
>> 
>> (add-hook 'after-make-frame-functions 'wg/setup-themes)
>> 
> 
> Note: only works with emacs24 themes.  You'd need a different
> theme-kludging function for color-theme-* themes, but same idea.
Could you expand on that ? 'color-theme-solarized-dark' is a function 
defined locally by solarized so I don't see how it is linked to emacs24.

Cheers,

---
Alexis Praga
GPG key : 66F9 4BD4 70EC E5D5 2AA4  9097 0673 7CE2 4671 306F