gnus: 'active' file for nnml:* getting out of sync

gnus: 'active' file for nnml:* getting out of sync

[Giorgos Keramidas]

I have noticed with a relatively recent trunk copy that if I compress
nnml folders with 'G z' of if I compact the entire nnml server in the
server buffer, by typing '^', then close and reopen Gnus a few times,
then new messages are marked as already 'old' ('O' marker) even if they
just arrived in the particular nnml folder.

After a bit of digging around I found that '~/Mail/active' does not
match what nnml thinks is inside the folder.  If there any way to force
an update of the nnml active file to match the in-memory state of the
nnml folder?

Re: gnus: 'active' file for nnml:* getting out of sync

[Giorgos Keramidas]

[-- Attachment #1.1: Type: text/plain, Size: 849 bytes --]

On Tue, 16 Mar 2010 07:14:48 +0200, Giorgos Keramidas <keramida@ceid.upatras.gr> wrote:
> I have noticed with a relatively recent trunk copy that if I compress
> nnml folders with 'G z' of if I compact the entire nnml server in the
> server buffer, by typing '^', then close and reopen Gnus a few times,
> then new messages are marked as already 'old' ('O' marker) even if they
> just arrived in the particular nnml folder.
>
> After a bit of digging around I found that '~/Mail/active' does not
> match what nnml thinks is inside the folder.  If there any way to force
> an update of the nnml active file to match the in-memory state of the
> nnml folder?

For example the attached PNG image shows my 'nnml:mail.mercurial.devel'
folder with messages that I have never seen before (they have the '.'
mark) marked already as 'old' at the same time:


[-- Attachment #1.2: gnus-desync.png --]
[-- Type: image/png, Size: 78010 bytes --]

[-- Attachment #1.3: Type: text/plain, Size: 142 bytes --]


Has anyone else seen this?  Is there any possibility my ~/.gnus file
causes this behavior?

A full copy of my .gnus file is attached below:


[-- Attachment #1.4: dot.gnus --]
[-- Type: text/plain, Size: 13195 bytes --]

;;; Gnus startup file.
;;
;; This file is not part of GNU Emacs.
;;
;; Copyright (C) 2002-2008, Giorgos Keramidas <keramida@freebsd.org>
;; All rights reserved.
;;
;; Redistribution and use in source and binary forms, with or without
;; modification, are permitted provided that the following conditions
;; are met:
;; 1. Redistributions of source code must retain the above copyright
;;    notice, this list of conditions and the following disclaimer.
;; 2. Redistributions in binary form must reproduce the above copyright
;;    notice, this list of conditions and the following disclaimer in the
;;    documentation and/or other materials provided with the distribution.
;;
;; THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
;; ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
;; IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
;; ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
;; FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
;; DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
;; OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
;; HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
;; LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
;; OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
;; SUCH DAMAGE.
;;
;; $GKer: .gnus a9b2c6f23884 2010/02/27 10:53:37 keramida $

;;; Code:

;; News and email source setup.
(setq gnus-nntp-server "news.sunsite.dk")
(setq nntp-authinfo-function 'nntp-send-authinfo)
(setq gnus-select-method '(nntp "news.sunsite.dk"))
(setq gnus-secondary-select-methods '((nnml "")))

;; For accessing Gmail messages, using imap+ssl, the following should work.
;(setq gnus-secondary-select-methods
;      '((nnml "")
;	(nnimap "gmail"
;		(nnimap-address "imap.gmail.com")
;		(nnimap-server-port 993)
;		(nnimap-stream ssl))))

;; Where to save a copy of all outgoing messages.
(setq gnus-message-archive-group "nnml:posted")

;; Save outgoing messages as 'read' by default.
(setq gnus-gcc-mark-as-read t)

;; Avoid html email, if at all possible.
(setq mm-discouraged-alternatives '("text/html"))

;; Default coding system to use for outgoing email.
(setq mm-coding-system-priorities '(iso-8859-1 utf-8))

;; Show buttonized UI elements for PGP signed messages.
(setq gnus-buttonized-mime-types
  '("multipart/alternative" "multipart/encrypted" "multipart/signed"
    ".*/signed"))

;; ---------------------------------------------------------------------
;; Message signing and signature verification through EasyPG.

(require 'epa)

(setq mml2015-use 'epg)
(setq mml2015-encrypt-to-self t)
(setq mml2015-verbose nil)
(setq mml2015-always-trust nil)
(setq mml2015-passphrase-cache-expiry '7200)

;; My default key for signing email messages.
(setq epg-user-id "318603B6")
(setq mml2015-signers (list "318603B6"))

(add-hook 'message-setup-hook 'keramida/get-mml2015-signers)
(defun keramida/get-mml2015-signers ()
  (when gnus-newsgroup-name
    (let ((signers (gnus-group-get-parameter gnus-newsgroup-name
					     'mml2015-signers t)))
      (when signers
	(set (make-local-variable 'mml2015-signers) signers)))))

(setq gnus-message-replysign t
      gnus-message-replyencrypt t
      gnus-message-replysignencrypted t
      gnus-treat-x-pgp-sig t
      mm-verify-option 'always
      mm-decrypt-option 'always)
;; ---------------------------------------------------------------------

;; Where my global score rules are saved.
(setq gnus-home-score-file
      "/home/keramida/News/gnus.SCORE")

;; Load nnmairix when Gnus fires up.  I like the way it can search very
;; fast through a great deal of email data.
(require 'nnmairix)

;; Mairix options & defaults.
(setq-default nnmairix-mairix-update-options (list "-F" "-p"))

;; Display and index buffer formats.
(require 'timezone)
(defun gnus-user-format-function-d (header)
  "Render article date in (YY-mm-dd) format."
    (let* ((date (mail-header-date header))
	     (tz (timezone-parse-date date)))
		 (format "%02d-%02d-%02d"
			 (string-to-int (aref tz 0))
			 (string-to-int (aref tz 1))
			 (string-to-int (aref tz 2)))))

(setq gnus-group-line-format "%M%S%8y: %G\n")
(setq gnus-summary-line-format "%U%R%z%ud %I%(%[%4L: %-20,20n%]%) %S\n")

;; How many faces to use for highlighting messages.  I get dizzy after
;; about three different faces, so keep their number small enough.
(setq gnus-cite-face-list '(gnus-cite-1 gnus-cite-2 gnus-cite-3))

;; Message reply & followup citation line format.
(defun gker-message-insert-citation-line ()
  "Insert a simple citation line."
  (when message-reply-headers
    (insert "On "
	    (mail-header-date message-reply-headers)
	    ", "
	    (mail-header-from message-reply-headers)
	    " wrote:")
    (newline)))
;; Bind my own citation line function, thus enabling it.
(setq message-citation-line-function 'gker-message-insert-citation-line)

;; Use only the text of the subject header to construct the default
;; subject of forwarded messages (instead of the Gnus default, that
;; includes the author name too).
(setq message-make-forward-subject-function 'message-forward-subject-fwd)

;; Prompt just once before saving many articles.
(setq gnus-prompt-before-saving t)

;; How to save articles, by default.
(setq gnus-default-article-saver 'gnus-summary-save-in-mail
      gnus-mail-save-name 'gnus-plain-save-name)

;; Use alt.foo.bar style names for folders
(setq gnus-use-long-file-name t)

;; When entering a folder, don't automagically select the first message
;; and show it in an article buffer.
(setq gnus-autoselect-first nil)

;; Display each article in it's own buffer (so that one can read several
;; groups at the same time).
(setq gnus-single-article-buffer nil)

;; The headers that are shown for messages or news-articles.
(setq gnus-visible-headers '("^Date:" "^From:" "^Reply-To:"
				"^Followup-To:" "^Subject:" "^To:"
				"^Cc:" "^Newsgroups:" "^Message-Id:"))
;; The order of headers shown in article buffers.
(setq gnus-sorted-header-list '("^Date:" "^From:" "^Reply-To:"
				"^Followup-To:" "^Subject:" "^To:"
				"^Cc:" "^Newsgroups:" "^Message-Id:"))

;; Customizations for article display.
(setq gnus-treat-date-local 'head)
(setq gnus-treat-display-face nil)
(setq gnus-treat-display-x-face nil)
(setq gnus-treat-display-smileys nil)

;; If displaying "text/html" is discouraged, images or other material
;; inside a "multipart/related" part might be overlooked when the
;; `gnus-mime-display-multipart-related-as-mixed' variable is `nil'.
(setq gnus-mime-display-multipart-related-as-mixed t)

;; Customizations for the Gnus Agent.
(setq gnus-agent-consider-all-articles nil)
(setq gnus-agent-cache t)
(setq gnus-use-cache t)
;(setq gnus-cacheable-groups "^nntp")

;;; Threading customizations.

(defun keramida-gnus-gather-threads-default (threads)
  "The default thread-gathering function for Gnus groups."
  (gnus-gather-threads-by-references threads))

(defvar keramida-gnus-group-thread-function-map
  '(("mail\\.freebsd\\.bugs" . gnus-gather-threads-by-subject)
    ("mail\\.freebsd\\.cvs\\.*" . gnus-gather-threads-by-references)
    ("mail\\.freebsd\\.core" . gnus-gather-threads-by-references))
  "Custom per-group mapping of Gnus group names to thread gathering functions.

The default value of this matches all the mail or news folders
that receive commit notifications in my personal Gnus
installation.  It is probably a good idea to edit the default
value in your `~/.gnus' file, and add the folders that makes
sense in your _own_ setup.")

(defun keramida-gnus-gather-threads (threads)
  "Dispatch function that matches Gnus group names to thread
gathering functions by looking up the group name in the
`keramida-gnus-group-thread-function-map'."
  (let ((group gnus-newsgroup-name))
    (let ((gather-function (or (and (fboundp 'keramida-gnus-gather-threads-default)
                                    (function keramida-gnus-gather-threads-default))
                               (and (fboundp 'gnus-gather-threads-by-references)
                                    (function gnus-gather-threads-by-references)))))
      (let ((match-function (and group
				 (boundp 'keramida-gnus-group-thread-function-map)
				 (assoc-default group keramida-gnus-group-thread-function-map
						'string-match))))
	(funcall (or match-function gather-function) threads)))))

;; Use my own thread-gathering function.
(setq gnus-summary-thread-gathering-function 'keramida-gnus-gather-threads)

;; Sort thread by date when in threaded mode, and articles by date again
;; when in unthreaded mode.
(setq gnus-thread-sort-functions '(gnus-thread-sort-by-date))
(setq gnus-article-sort-functions '(gnus-article-sort-by-number gnus-article-sort-by-date))

;; BBDB support for the Big Brother database.
(require 'bbdb)
(bbdb-initialize 'gnus 'message)

;; Mode hooks
(add-hook 'gnus-group-mode-hook 'gnus-topic-mode)

;; Swap the default behavior of the 'e' and 'E' keys in the group summary
;; buffers.  Using a shifted key to expire articles is somewhat painful.  I
;; expire articles far too often, but I only very occasionally edit the
;; contents of email articles.  So it should be easier to expire them.
(add-hook 'gnus-summary-mode-hook 'keramida-alter-summary-map)
(add-hook 'gnus-article-prepare-hook 'keramida-alter-summary-map)
(defun keramida-alter-summary-map ()
  ;; The `gnus-summary-put-mark-as-expirable-next' command can work with
  ;; a prefix arg too, so we wrap it in an interactive lambda.  It also
  ;; expects `gnus-summary-buffer' to be the current buffer, otherwise
  ;; it fails to work from *Article* buffers.
  (local-set-key "e" "MMen")
  (when nil
    (local-set-key "e"
      (lambda (n)
	(interactive "p")
	(with-current-buffer gnus-summary-buffer
	  (gnus-summary-put-mark-as-expirable-next n)))))
  ;; Article editing is slightly simpler.  It's designed to work fine
  ;; when called from article buffers, so we don't need to do anything
  ;; special to invoke it.
  (local-set-key "E" 'gnus-summary-edit-article))

;;; Mail expiry and archival:
;;

(setq nnmail-expiry-target 'nnmail-fancy-expiry-target
      nnmail-fancy-expiry-targets '(("from" ".*" "nnml:mail.archive-%Y")))

;;; Mail splitting with Gnus:
;;

(setq nnmail-split-methods 'nnmail-split-fancy)
(setq nnmail-split-fancy

      '(| ;; Save messages tagged as spam to the "mail.junk" folder, to
          ;; be able to review the list later for false-positives.

	  ("x-spam-flag" "yes" "mail.junk")
	  ("subject" ".*{Spam not delivered}.*" "mail.junk")
	  ("subject" ".*{Possible Spam}.*" "mail.junk")

	  ;; Throw away messages from blacklisted people.

	  (from "drososkourounis@yahoo\\.gr" junk)
	  (from "zakinthinos@freemail\\.gr" junk)

	  ;; FreeBSD email filters.  The `core' and `developers' mailboxes are
	  ;; treated in a special way, so that I won't lose email posted
	  ;; there in the noise of a mailing list.

	  (any "freebsd\\.org"

	       (| (& ("delivered-to" "core@.*freebsd\\.org" "mail.freebsd.core")
		     ("delivered-to" "doceng@.*freebsd\\.org" "mail.freebsd.doceng")
		     ("delivered-to" "developers@.*freebsd\\.org" "mail.freebsd.misc")

		     ("sender" "owner-doc-committers@freebsd\\.org" "mail.freebsd.cvs.doc")
		     ("sender" "owner-cvs-\\(\\w+\\)@.*freebsd\\.org" "mail.freebsd.cvs.\\1")
		     ("sender" "owner-svn-src-\\(\\w+\\)@.*freebsd\\.org" "mail.freebsd.svn.src.\\1")
		     ("sender" "owner-freebsd-\\(\\w+\\)@.*freebsd\\.org" "mail.freebsd.\\1"))

		  ;; Fallback folder for all the other FreeBSD messages.
		  "mail.freebsd.misc"))

	  ;; Bit bucket issue tracker emails.
	  (from "issues-noreply@bitbucket.org" "mail.bitbucket.issues")

	  ;; Facebook email
	  (any ".*@.*facebook\\.com" "mail.facebook")

	  ;; GNU mailing lists.
	  ("sender" "emacs-devel.*@gnu\\.org" "mail.gnu.emacs.devel")
	  ("list-id" "ding\\.gnus\\.org" "mail.gnu.gnus.ding")
	  ("list-id" "erc-discuss\\.gnu\\.org" "mail.gnu.erc.discuss")

	  ;; Greek GRNET / Academic Network lists.
	  ("list-id" "epistimoniki-epitropi-ellak\\.grnet\\.gr" "mail.grnet.epistimoniki-epitropi-ellak")
	  ("list-id" "open-source\\.ellak\\.gr" "mail.grnet.open-source")

	  ;; HELLUG mailing lists.
	  ("list-id" "freebsd-doc-el\\.lists\\.hellug\\.gr" "mail.freebsd.doc.el")
	  ("list-id" "linux-greek-users\\.lists\\.hellug\\.gr" "mail.linux.greek.users")
	  ("list-id" "migrate2linux\\.lists\\.hellug\\.gr" "mail.linux.greek.migrate")
	  ("list-id" "plug\\.lists\\.hellug\\.gr" "mail.linux.greek.plug")

	  ;; Mercurial mailing lists.
	  ("sender" "mercurial-bounces@selenic\\.com" "mail.mercurial")
	  ("sender" "mercurial-devel-bounces@selenic\\.com" "mail.mercurial.devel")

	  ;; TeX, LaTeX and ConTeXt mailing lists.
	  ("list-id" "ntg-context\\.ntg\\.nl" "mail.tex.context")

	  ;; FFII mailing lists.
	  ("list-id" "news\\.ffii\\.org" "mail.ffii-news")

	  ;; Finally, the fallback folder for all messages that are *still*
	  ;; undelivered, is the main `INBOX'.

	  "mail.INBOX"))

;;; Local variables:
;;; mode: emacs-lisp
;;; coding: utf-8
;;; fill-column: 72
;;; End:

[-- Attachment #2: Type: application/pgp-signature, Size: 195 bytes --]

Re: gnus: 'active' file for nnml:* getting out of sync

[Reiner Steib]

On Tue, Mar 16 2010, Giorgos Keramidas wrote:

> I have noticed with a relatively recent trunk copy that if I compress
> nnml folders with 'G z' of if I compact the entire nnml server in the
> server buffer, by typing '^', then close and reopen Gnus a few times,
> then new messages are marked as already 'old' ('O' marker) even if they
> just arrived in the particular nnml folder.

Seems like a bug in the nnml compaction feature.  Didier?

> After a bit of digging around I found that '~/Mail/active' does not
> match what nnml thinks is inside the folder.  If there any way to force
> an update of the nnml active file to match the in-memory state of the
> nnml folder?

There are `nnml-generate-nov-databases' and
`nnml-generate-nov-databases-directory'.

,----[ (info "(gnus)Mail Spool") ]
|    If your `nnml' groups and NOV files get totally out of whack, you
| can do a complete update by typing `M-x nnml-generate-nov-databases'.
| This command will trawl through the entire `nnml' hierarchy, looking at
| each and every article, so it might take a while to complete.  A better
| interface to this functionality can be found in the server buffer
| (*note Server Commands::).
`----

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: gnus: 'active' file for nnml:* getting out of sync

[Didier Verna]

Reiner Steib <reinersteib+gmane@imap.cc> wrote:

> On Tue, Mar 16 2010, Giorgos Keramidas wrote:
>
>> I have noticed with a relatively recent trunk copy that if I compress
>> nnml folders with 'G z' of if I compact the entire nnml server in the
>> server buffer, by typing '^', then close and reopen Gnus a few times,
>> then new messages are marked as already 'old' ('O' marker) even if they
>> just arrived in the particular nnml folder.
>
> Seems like a bug in the nnml compaction feature.  Didier?

Been a while since I touched it (I'm using nnimap now) but I'll try to
look at it.

Giorgos, what do you mean by "new": do you mean unread messages that are
here before compaction, or messages that arrive afterwards ?

-- 
Resistance is futile. You will be jazzimilated.

Scientific site:   http://www.lrde.epita.fr/~didier
Music (Jazz) site: http://www.didierverna.com

Re: gnus: 'active' file for nnml:* getting out of sync

[Giorgos Keramidas]

On Tue, 16 Mar 2010 14:28:51 +0100, Didier Verna <didier@xemacs.org> wrote:
> Reiner Steib <reinersteib+gmane@imap.cc> wrote:
>
>> On Tue, Mar 16 2010, Giorgos Keramidas wrote:
>>
>>> I have noticed with a relatively recent trunk copy that if I compress
>>> nnml folders with 'G z' of if I compact the entire nnml server in the
>>> server buffer, by typing '^', then close and reopen Gnus a few times,
>>> then new messages are marked as already 'old' ('O' marker) even if they
>>> just arrived in the particular nnml folder.
>>
>> Seems like a bug in the nnml compaction feature.  Didier?
>
> Been a while since I touched it (I'm using nnimap now) but I'll try to
> look at it.
>
> Giorgos, what do you mean by "new": do you mean unread messages that are
> here before compaction, or messages that arrive afterwards ?

Messages that arrive afterwards.  The steps I have to follow to
reproduce this are:

  1. Compact an nnml folder with 'G z'.
  2. Get new messages with 'g'.

I would expect the newly arrived messages to be marked as '.' when I
enter the folder and not be marked as 'old'.  But they are.