Gnus with Emacs 23: gnus-gcc-mark-as-read not working anymore

Gnus with Emacs 23: gnus-gcc-mark-as-read not working anymore

[Stefan Kamphausen]

Hello,

last week Emacs 23.1 became stable on Gentoo and so I upgraded.  Mostly
smoothly.

However, there is one annoying problem which is that
gnus-gcc-mark-as-read doesn't seem to work anymore.

As far as I can see the relevant sections of my gnus-init-file are

(setq gnus-message-archive-group
      '((cond ((message-news-p)
               "nnml:saved-news")
              (t
               (list
                "nnml:mail.office.all"
                (format-time-string "nnml:mail.%Y-%m") )))))
(setq gnus-gcc-mark-as-read t)

The GCC-header is set all right, the messages are archived, too, but
they are marked as unread and I need to catchup manually.  This was not
the case with Emacs 22.x.

I already tried to dig into the code and already asked the same question
in a German gnus newsgroup:

http://groups.google.de/group/de.comm.software.gnus/browse_thread/thread/cf09458e0ea65b9b#

but to no avail.

Can someone please give any hints?  Given the amount of time I already
spent on this, the solution should be something quite obvious that I
just don't see.


Thanks and cheers,
Stefan


PS:
emacs-version "23.1.1"
gnus-version "Gnus v5.13" (as shipped with Emacs)

-- 
Stefan Kamphausen --- http://www.skamphausen.de
a blessed +42 regexp of confusion (weapon in hand)
You hit. The format string crumbles and turns to dust.

Re: Gnus with Emacs 23: gnus-gcc-mark-as-read not working anymore

[Stefan Kamphausen]

Hi all,

in the meantime I dived into the code and found that my description was
not fully correct.  Right after sending a message with GCC-headers those
messages seem to be marked read, because they don't show up immediately
after sending.  Just when I hit 'g' (gnus-group-get-new-news) the new
message make the group appear bold and with a message count > 0 in the
group buffer.

Without really understanding the code, is it possible that this happens
because the agent will regenerate the group first and the messages will
be marked as read afterwards?

Setting gnus-verbose to a value >10 and by adding 
  (message "DEBUG Mark as read! %s %s" group (cdr group-art))
the function gnus-inews-do-gcc just before
  (gnus-group-mark-article-read group (cdr group-art))
I get the following in the message buffer:


Sending...
Sending via mail...
Wrote /home/user/contact/gnus/mail/office/all/31319
Regenerating in nnml:mail.office.all
DEBUG Mark as read! mail.office.all 31319
Wrote /home/user/contact/gnus/mail/2009-09/303
Regenerating in nnml:mail.2009-09
DEBUG Mark as read! mail.2009-09 303
Sending...done


Any ideas?  Or maybe is there a better place to ask?


Kind regards,
Stefan
-- 
Stefan Kamphausen --- http://www.skamphausen.de
a blessed +42 regexp of confusion (weapon in hand)
You hit. The format string crumbles and turns to dust.

Re: Gnus with Emacs 23: gnus-gcc-mark-as-read not working anymore

[Stefan Kamphausen]

Hi.


Stefan Kamphausen <skampi@gmx.net> writes:

> Hi all,
>
> in the meantime I dived into the code and found that my description was
> not fully correct.  Right after sending a message with GCC-headers those
> messages seem to be marked read, because they don't show up immediately
> after sending.  Just when I hit 'g' (gnus-group-get-new-news) the new
> message make the group appear bold and with a message count > 0 in the
> group buffer.
>
> Without really understanding the code, is it possible that this happens
> because the agent will regenerate the group first and the messages will
> be marked as read afterwards?
>
> Setting gnus-verbose to a value >10 and by adding 
>   (message "DEBUG Mark as read! %s %s" group (cdr group-art))
> the function gnus-inews-do-gcc just before
>   (gnus-group-mark-article-read group (cdr group-art))
> I get the following in the message buffer:
>
>
> Sending...
> Sending via mail...
> Wrote /home/user/contact/gnus/mail/office/all/31319
> Regenerating in nnml:mail.office.all
> DEBUG Mark as read! mail.office.all 31319
> Wrote /home/user/contact/gnus/mail/2009-09/303
> Regenerating in nnml:mail.2009-09
> DEBUG Mark as read! mail.2009-09 303
> Sending...done

Rereading my older post and while searching the gnus-source code as well
as the internet for answers I found that the method is stripped from the
group name.  According to this thread from 2006
http://thread.gmane.org/gmane.emacs.gnus.general/62775/focus=62934
this may well be the problem.

To me it seems like the call to (gnus-group-short-name group) leads to
my problem.  If I read the changelog correctly it was introduced in
revision 1.57, Fri Jan 9 03:01:51 2009 UTC:

http://cvs.savannah.gnu.org/viewvc/emacs/lisp/gnus/gnus-msg.el?root=emacs&r1=1.56&r2=1.57


Will I have to switch to nnfolder+archive for archiving my messages?
That would be a pity because I archive to a group which I also use to
receive (all) messages to.  Do you know of any way to still keep the
nnml:prefix?  The documentation says that I can do so:
http://gnus.org/manual/gnus_152.html#SEC152


Any help highly appreciated.

Kind regards,
Stefan
-- 
Stefan Kamphausen --- http://www.skamphausen.de
a blessed +42 regexp of confusion (weapon in hand)
You hit. The format string crumbles and turns to dust.

Re: Gnus with Emacs 23: gnus-gcc-mark-as-read not working anymore

[David Engster]

Stefan Kamphausen <skampi@gmx.net> writes:
> Stefan Kamphausen <skampi@gmx.net> writes:
>> in the meantime I dived into the code and found that my description was
>> not fully correct.  Right after sending a message with GCC-headers those
>> messages seem to be marked read, because they don't show up immediately
>> after sending.  Just when I hit 'g' (gnus-group-get-new-news) the new
>> message make the group appear bold and with a message count > 0 in the
>> group buffer.

Does hitting M-g on the group fix this?

> Rereading my older post and while searching the gnus-source code as well
> as the internet for answers I found that the method is stripped from the
> group name.  According to this thread from 2006
> http://thread.gmane.org/gmane.emacs.gnus.general/62775/focus=62934
> this may well be the problem.
>
> To me it seems like the call to (gnus-group-short-name group) leads to
> my problem.  If I read the changelog correctly it was introduced in
> revision 1.57, Fri Jan 9 03:01:51 2009 UTC:
>
> http://cvs.savannah.gnu.org/viewvc/emacs/lisp/gnus/gnus-msg.el?root=emacs&r1=1.56&r2=1.57

This was introduced because of a problem with nnimap as primary mail
backend. See

http://thread.gmane.org/gmane.emacs.gnus.general/67932

Do you use nnml as your primary back end?

Regards,
David

Re: Gnus with Emacs 23: gnus-gcc-mark-as-read not working anymore

[Stefan Kamphausen]

Hi David,

David Engster <deng@randomsample.de> writes:

> Stefan Kamphausen <skampi@gmx.net> writes:
>> Stefan Kamphausen <skampi@gmx.net> writes:
>>> in the meantime I dived into the code and found that my description was
>>> not fully correct.  Right after sending a message with GCC-headers those
>>> messages seem to be marked read, because they don't show up immediately
>>> after sending.  Just when I hit 'g' (gnus-group-get-new-news) the new
>>> message make the group appear bold and with a message count > 0 in the
>>> group buffer.
>
> Does hitting M-g on the group fix this?

I'm not sure what you mean by "fix".  However, if I go to one of my
archive groups (nnml:mail.2009-09 currently) and hit M-x the messages
sent earlier lead to that group having a count > 0, that is, those sent
messages are not marked as read.  So, this is just the same as hitting
'g' anywhere in the group buffer.


>> Rereading my older post and while searching the gnus-source code as well
>> as the internet for answers I found that the method is stripped from the
>> group name.  According to this thread from 2006
>> http://thread.gmane.org/gmane.emacs.gnus.general/62775/focus=62934
>> this may well be the problem.
>>
>> To me it seems like the call to (gnus-group-short-name group) leads to
>> my problem.  If I read the changelog correctly it was introduced in
>> revision 1.57, Fri Jan 9 03:01:51 2009 UTC:
>>
>> http://cvs.savannah.gnu.org/viewvc/emacs/lisp/gnus/gnus-msg.el?root=emacs&r1=1.56&r2=1.57
>
> This was introduced because of a problem with nnimap as primary mail
> backend. See
>
> http://thread.gmane.org/gmane.emacs.gnus.general/67932
>
> Do you use nnml as your primary back end?

Yes:

(setq gnus-select-method '(nnml ""))
(setq gnus-secondary-select-methods
      '(
      ;; Had lots of nnimap groups here earlier, but now all are commented
      ;; However, I might need them again in the future
      ))

(eval-after-load "mail-source"
  '(setq mail-sources
         '(
           (imap
            :server "mailserver.example.com"
            :user "domain/username"
            :password "verysecret"
            )
           )))


I think I could switch primary to nnil and secondary to nnml, but I am
not sure.  

Is it dangerous to try this?  Would it mess up some files, like
~/.news.eld?  What would I need to backup before any experiments?


Kind Regards,
Stefan
-- 
Stefan Kamphausen --- http://www.skamphausen.de
a blessed +42 regexp of confusion (weapon in hand)
You hit. The format string crumbles and turns to dust.

Re: Gnus with Emacs 23: gnus-gcc-mark-as-read not working anymore

[David Engster]

Stefan Kamphausen <skampi@gmx.net> writes:
> David Engster <deng@randomsample.de> writes:
>> Stefan Kamphausen <skampi@gmx.net> writes:
>>> Stefan Kamphausen <skampi@gmx.net> writes:
>>>> in the meantime I dived into the code and found that my description was
>>>> not fully correct.  Right after sending a message with GCC-headers those
>>>> messages seem to be marked read, because they don't show up immediately
>>>> after sending.  Just when I hit 'g' (gnus-group-get-new-news) the new
>>>> message make the group appear bold and with a message count > 0 in the
>>>> group buffer.
>>
>> Does hitting M-g on the group fix this?
>
> I'm not sure what you mean by "fix".

I meant if it then corrects the number of unread messages to "0". This
could happen if you have 'gaps' in the article numbers, which can often
be resolved by doing a M-g on that group. But that doesn't seem to be
the case here.

>> This was introduced because of a problem with nnimap as primary mail
>> backend. See
>>
>> http://thread.gmane.org/gmane.emacs.gnus.general/67932
>>
>> Do you use nnml as your primary back end?
>
> Yes:
>
> (setq gnus-select-method '(nnml ""))
> (setq gnus-secondary-select-methods
>       '(
>       ;; Had lots of nnimap groups here earlier, but now all are commented
>       ;; However, I might need them again in the future
>       ))

[...]

> I think I could switch primary to nnil and secondary to nnml, but I am
> not sure.  
>
> Is it dangerous to try this?  Would it mess up some files, like
> ~/.news.eld?  What would I need to backup before any experiments?

No, better don't do this. It should work with nnml as primary back end,
otherwise it's a bug. I don't use nnml, so I'm not sure why it doesn't
work for you. Maybe nnml and nnimap behave differently in this regard,
which would be unfortunate. I will have to create a test setup for this.

Regards,
David

Re: Gnus with Emacs 23: gnus-gcc-mark-as-read not working anymore

[Stefan Kamphausen]

Hi David,

first of all: thank you for your time and effort.

David Engster <deng@randomsample.de> writes:

>>> This was introduced because of a problem with nnimap as primary mail
>>> backend. See
>>>
>>> http://thread.gmane.org/gmane.emacs.gnus.general/67932
>>>
>>> Do you use nnml as your primary back end?
>>
>> Yes:
>>
>> (setq gnus-select-method '(nnml ""))
>> (setq gnus-secondary-select-methods
>>       '(
>>       ;; Had lots of nnimap groups here earlier, but now all are commented
>>       ;; However, I might need them again in the future
>>       ))
>
> [...]
>
>> I think I could switch primary to nnil and secondary to nnml, but I am
>> not sure.  
>>
>> Is it dangerous to try this?  Would it mess up some files, like
>> ~/.news.eld?  What would I need to backup before any experiments?
>
> No, better don't do this. It should work with nnml as primary back end,
> otherwise it's a bug. I don't use nnml, so I'm not sure why it doesn't
> work for you. Maybe nnml and nnimap behave differently in this regard,
> which would be unfortunate. I will have to create a test setup for this.

Currently I am manually catching up all the mails in all archive groups,
so any improvement of this situation will make my life happier ;-)

If I can be of any help for that test setup, please let me know.  I
/think/ it would be sufficient to create the Gcc-header manually such as
for this message here:

Gcc: nnml:saved-news

and then sent a message to yourself.  I will happily test all the
functions, that you may send, manually, if this helps.  I could narrow
the problem down to line 1685 in gnus-msg.el:

		(setq group (gnus-group-short-name group)))

since gnus-group-short-name strips the nnml-prefix.  I just evaluated
the gnus-inews-do-gcc defun found in Emacs 22.3 and then everything
worked as expected.


Kind regards,
Stefan
-- 
Stefan Kamphausen --- http://www.skamphausen.de
a blessed +42 regexp of confusion (weapon in hand)
You hit. The format string crumbles and turns to dust.

Re: Gnus with Emacs 23: gnus-gcc-mark-as-read not working anymore

[David Engster]

Stefan Kamphausen <skampi@gmx.net> writes:
> David Engster <deng@randomsample.de> writes:
>>>> This was introduced because of a problem with nnimap as primary mail
>>>> backend. See
>>>>
>>>> http://thread.gmane.org/gmane.emacs.gnus.general/67932
>>>>
>>>> Do you use nnml as your primary back end?
>>>
>>> Yes

> I could narrow the problem down to line 1685 in gnus-msg.el:
>
> 		(setq group (gnus-group-short-name group)))
>
> since gnus-group-short-name strips the nnml-prefix.  I just evaluated
> the gnus-inews-do-gcc defun found in Emacs 22.3 and then everything
> worked as expected.

OK, I tested this with Emacs 23.1/Gnus v5.13 with the following minimal
.gnus.el:

(setq gnus-select-method '(nnml ""))
(setq gnus-message-archive-group
      '((cond ((message-news-p)
               "nnml:saved-news")
              (t
               (list
                "nnml:mail.office.all"
                (format-time-string "nnml:mail.%Y-%m") )))))
(setq gnus-gcc-mark-as-read t)

It works for me. I get the following GCC header

Gcc: nnml:mail.office.all nnml:mail.2009-09

and when I send mail, it is marked as read and doesn't show up when I
hit 'g'.

In fact, when I comment out the call to gnus-group-short-name, it does
not work anymore and I get a new article in both groups.

I don't know why you're seeing the exact opposite behavior. Please check
if the articles actually get marked as read. You can do this by
evaluating first

(gnus-active "mail.office.all")

which will give you the lowest and highest article number, and

(gnus-info-read (gnus-get-info "mail.office.all"))

which will give you the range of read articles. You should get something
like (1 . 10) and ((1 . 10)), resp., with '10' being the highest article
number in my case.

Then send a message and immediately evaluate the above again. It should
be something like (1 . 11) and ((1 . 11)).

Regards,
David

Re: Gnus with Emacs 23: gnus-gcc-mark-as-read not working anymore

[Stefan Kamphausen]

Hi David,

sorry for the delay.

David Engster <deng@randomsample.de> writes:

> OK, I tested this with Emacs 23.1/Gnus v5.13 with the following minimal
> .gnus.el:
>

[...]

> It works for me. I get the following GCC header
>
> Gcc: nnml:mail.office.all nnml:mail.2009-09
>
> and when I send mail, it is marked as read and doesn't show up when I
> hit 'g'.
>
> In fact, when I comment out the call to gnus-group-short-name, it does
> not work anymore and I get a new article in both groups.

Interesting.  I just deinstalled Emacs 22 which was installed in
parallel to Emacs 23, which is possible on Gentoo, to exclude problems
with that.  To no avail.

> I don't know why you're seeing the exact opposite behavior. Please check
> if the articles actually get marked as read. You can do this by
> evaluating first
>
> (gnus-active "mail.office.all")
>
> which will give you the lowest and highest article number, and
>
> (gnus-info-read (gnus-get-info "mail.office.all"))
>
> which will give you the range of read articles. You should get something
> like (1 . 10) and ((1 . 10)), resp., with '10' being the highest article
> number in my case.
>
> Then send a message and immediately evaluate the above again. It should
> be something like (1 . 11) and ((1 . 11)).

I did as you suggested and the results are in the following transcript
of the *scratch*-buffer with lots of C-j's hit after the closing parens
to evaluate the code.

;; Before sending a mail
;; Please note that there are always messages in mail.office.all
;; marked as unread.  I use this to mark messages needing attention (I
;; also use ! for another purpose)
(gnus-active "mail.office.all")
(3 . 31806)

(gnus-info-read (gnus-get-info "mail.office.all"))
nil

(gnus-info-read (gnus-get-info "nnml:mail.office.all")) 
((1 . 28685) (28687 . 28971) (28974 . 30386) (30388 . 30399) (30401
. 30498) (30500 . 30849) (30851 . 31339) (31341 . 31504) (31506 . 31599)
(31601 . 31608) (31610 . 31636) (31638 . 31662) ...)

;; The other gcc group
(gnus-active "nnml:mail.2009-10")
(1 . 25)

(gnus-info-read (gnus-get-info "nnml:mail.2009-10"))
((1 . 25))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Now sending a mail to myself with Gcc-headers like this:
;; Gcc: nnml:mail.office.all nnml:mail.2009-10
(gnus-active "mail.office.all")
(3 . 31807)

(gnus-active "nnml:mail.2009-10")
(1 . 25) ; strange?

(gnus-info-read (gnus-get-info "nnml:mail.2009-10"))
((1 . 25))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Now getting mail by hitting 'g' in the Group buffer.  Please note,
;; that I deleted the mail I sent to myself via IMAP with another
;; mailclient to not interfer with results here
(gnus-active "mail.office.all")
(3 . 31807)

(gnus-active "nnml:mail.2009-10")
(1 . 26) ; stranger??

(gnus-info-read (gnus-get-info "nnml:mail.2009-10"))
((1 . 25)) ; strangest???

;; Both groups (mail.office.all and mail.2009-10) now show one more
;; message than before.


I would like to try with emacs -Q and a minimal gnus config file,
however, since I don't really know what is going on behind the scenes
(like .news.eld) I don't see how I could do that without interfering
with my production email setup.


Kind regards,
Stefan
-- 
Stefan Kamphausen --- http://www.skamphausen.de
a blessed +42 regexp of confusion (weapon in hand)
You hit. The format string crumbles and turns to dust.

Re: Gnus with Emacs 23: gnus-gcc-mark-as-read not working anymore

[David Engster]

Stefan Kamphausen <skampi@gmx.net> writes:
> ;; Before sending a mail
[...]
> ;; The other gcc group
> (gnus-active "nnml:mail.2009-10")
> (1 . 25)
>
> (gnus-info-read (gnus-get-info "nnml:mail.2009-10"))
> ((1 . 25))
>
> ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
> ;; Now sending a mail to myself with Gcc-headers like this:
> ;; Gcc: nnml:mail.office.all nnml:mail.2009-10
[...]
> (gnus-active "nnml:mail.2009-10")
> (1 . 25) ; strange?
>
> (gnus-info-read (gnus-get-info "nnml:mail.2009-10"))
> ((1 . 25))

It seems Gnus did not notice yet that a new mail is in in
mail.2009-10. Since I cannot reproduce this, I don't know why that
happens. You mentioned that you're using the agent; maybe that's the
cause of this issue.

If you want to further debug this, the best place to start is
gnus-inews-do-gcc, where the actual sending happens with the call to
gnus-request-accept-article here:

	      (when (or (not (gnus-check-backend-function
			      'request-accept-article group))
			(not (setq group-art
				   (gnus-request-accept-article
				    group method t t))))
		(gnus-message 1 "Couldn't store article in group %s: %s"
			      group (gnus-status-message method)))

Maybe a

(gnus-request-group group nil method)

after the sending will force Gnus to update the active information. I'm
not sure.

Regards,
David

Re: Gnus with Emacs 23: gnus-gcc-mark-as-read not working anymore

[Stefan Kamphausen]

Hi David,

David Engster <deng@randomsample.de> writes:

> It seems Gnus did not notice yet that a new mail is in in
> mail.2009-10. Since I cannot reproduce this, I don't know why that
> happens. You mentioned that you're using the agent; maybe that's the
> cause of this issue.

I'm not doing so consciously, I never fiddled with any agent
settings. The only setting concerning Gnus agent I have in my
configuration is
(setq gnus-agent-enable-expiration 'DISABLE)


> If you want to further debug this, the best place to start is
> gnus-inews-do-gcc, where the actual sending happens with the call to
> gnus-request-accept-article here:
> 	      (when (or (not (gnus-check-backend-function
> 			      'request-accept-article group))
> 			(not (setq group-art
> 				   (gnus-request-accept-article
> 				    group method t t))))
> 		(gnus-message 1 "Couldn't store article in group %s: %s"
> 			      group (gnus-status-message method)))

I always looked a little bit further down the code:

              (when (and (listp method)
                         (gnus-native-method-p method))
                (setq group (gnus-group-short-name group)))
              (when (and group-art
                         ;; FIXME: Should gcc-mark-as-read work when
                         ;; Gnus is not running?
                         (gnus-alive-p)
                         (or gnus-gcc-mark-as-read
                             (and
                              (boundp 'gnus-inews-mark-gcc-as-read)
                              (symbol-value 'gnus-inews-mark-gcc-as-read))))
                (gnus-group-mark-article-read group (cdr group-art)))


And when I change 
                (setq group (gnus-group-short-name group)))
to just
                (setq group group))
(for the moment) everything works as expected.

I tried to trace the functions

 * gnus-group-mark-article-read 
 * gnus-mark-article-as-read
 * gnus-group-make-articles-read

with the following result:
;; the following call DID NOT work as expected
======================================================================
1 -> gnus-group-mark-article-read: group="mail.office.all" article=31891
| 2 -> gnus-group-make-articles-read: group="mail.office.all" articles=(31891)
| 2 <- gnus-group-make-articles-read: nil
1 <- gnus-group-mark-article-read: nil
======================================================================
1 -> gnus-group-mark-article-read: group="mail.2009-10" article=65
| 2 -> gnus-group-make-articles-read: group="mail.2009-10" articles=(65)
| 2 <- gnus-group-make-articles-read: nil
1 <- gnus-group-mark-article-read: nil

;; the following call DID work as expected
======================================================================
1 -> gnus-group-mark-article-read: group="nnml:mail.office.all" article=31892
| 2 -> gnus-group-make-articles-read: group="nnml:mail.office.all" articles=(31892)
| 2 <- gnus-group-make-articles-read: #<buffer  *acc*>
1 <- gnus-group-mark-article-read: nil
======================================================================
1 -> gnus-group-mark-article-read: group="nnml:mail.2009-10" article=66
| 2 -> gnus-group-make-articles-read: group="nnml:mail.2009-10" articles=(66)
| 2 <- gnus-group-make-articles-read: #<buffer  *acc*>
1 <- gnus-group-mark-article-read: nil

> (gnus-request-group group nil method)
>
> after the sending will force Gnus to update the active information. I'm
> not sure.

Hm, I changed that part of gnus-inews-do-gcc to read

               (when (or (not (gnus-check-backend-function
                              'request-accept-article group))
                        (not (setq group-art
                                   (gnus-request-accept-article
                                    group method t t))))
                (gnus-message 1 "Couldn't store article in group %s: %s"
                              group (gnus-status-message method)))
              (gnus-request-group group nil method)

and it doesn't help.


I think, I'll have to live with my own version of gnus-inews-do-gcc
which does not use the short name of the group.  At least, *I* do not
see anything that I could do to examine this any further.



Thanks and kind regards,
Stefan
-- 
Stefan Kamphausen --- http://www.skamphausen.de
a blessed +42 regexp of confusion (weapon in hand)
You hit. The format string crumbles and turns to dust.