bug#53352: 29.0.50; IMAP INBOX not refreshing using `gnus-group-get-new-news'

bug#53352: 29.0.50; IMAP INBOX not refreshing using `gnus-group-get-new-news'

[Michael Welsh Duggan]

I have a gnus installation where the select method is as such:

    ;;; Select Method
    (setq gnus-select-method
          '(nnimap "localimap"
                    (nnimap-inbox "INBOX")
                    (nnimap-address "xxxxxxxx.xxx")))

The IMAP server is Dovecot 2.3.17.1.  There's nothing special in the
group parameters for the INBOX folder, currently this:

    ((modseq . "6496")
     (uidvalidity . "1374073954")
     (active 1 . 2140)
     (permanent-flags %Answered %Flagged %Deleted %Seen %Draft gnus-forward gnus-download %*))

The group activation level is 1.

Mail splitting is currently being handled by sieve, not by Gnus.
However, when sieve leaves a message in INBOX rather than sorting it to
another folder, it does not appear in INBOX using `g'
(gnus-group-get-new-news).  Any mail that stays there does show up if I
use `M-g' (gnus-topic-get-new-news-this-topic) on the group.  This only
happens for the INBOX folder.  I don't believe INBOX should behave
differently from other folders in this regard.

In GNU Emacs 29.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo version 1.16.0)
 of 2022-01-12 built on miko
Repository revision: f2fcea2716405aac9a3a6013b284b6b7be634725
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12013000
System Description: Debian GNU/Linux bookworm/sid

Configured using:
 'configure --without-toolkit-scroll-bars --with-x-toolkit=lucid
 --with-native-compilation --with-xinput2'

Configured features:
CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG JSON
LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES NATIVE_COMP
NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF X11
XAW3D XDBE XIM XINPUT2 XPM LUCID ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Load-path shadows:
/home/md5i/.config/emacs/elpa/transient-20211105.100/transient hides /usr/local/share/emacs/29.0.50/lisp/transient

Features:
(shadow emacsbug dabbrev shr-color color sieve-mode org-element avl-tree
generator ol-eww eww xdg url-queue mm-url ol-rmail ol-mhe ol-irc ol-info
ol-gnus nnselect gnus-search eieio-opt speedbar ezimage dframe
ol-docview doc-view ol-bibtex ol-bbdb ol-w3m ol-doi org-link-doi org ob
ob-tangle ob-ref ob-lob ob-table ob-exp org-macro org-footnote org-src
ob-comint org-pcomplete org-list org-faces org-entities noutline outline
org-version ob-emacs-lisp ob-core ob-eval org-table oc-basic bibtex ol
org-keys oc org-compat advice org-macs org-loaddefs find-func cal-menu
calendar cal-loaddefs grep copyright time-stamp whitespace mailalias
smtpmail sendmail help-fns radix-tree magit-patch magit-subtree
magit-gitignore magit-ediff magit-extras calc-alg calc-ext calc-menu
calc calc-loaddefs rect calc-macs misearch multi-isearch tabify man
generic sh-script smie executable goto-addr vc-mtn vc-hg vc-bzr vc-src
vc-sccs vc-svn vc-cvs vc-rcs vc bug-reference face-remap thingatpt
vc-git vc-dispatcher flow-fill gnus-fun jka-compr image-mode exif
dired-aux mule-util sort gnus-cite mm-archive mail-extr qp gnus-async
gnus-bcklg gnus-ml disp-table gnus-topic cursor-sensor nndraft nnmh
nnfolder utf-7 epa-file network-stream gnus-agent gnus-srvr gnus-score
score-mode nnvirtual gnus-msg gnus-art mm-uu mml2015 mm-view mml-smime
smime dig gnus-cache gnus-sum shr pixel-fill kinsoku svg gnus-demon nntp
gnus-group gnus-undo gnutls gnus-start gnus-dbus gnus-cloud gnus-spec
gnus-win flyspell ispell view pacproxy descr-text cus-edit pp cus-load
time sieve-manage sasl sasl-anonymous sasl-login sasl-plain rng-loc
rng-uri rng-parse rng-match rng-dt rng-util rng-pttrn nxml-parse nxml-ns
nxml-enc xmltok nxml-util sgml-mode facemenu dom python tramp-sh tramp
tramp-loaddefs trampver tramp-integration files-x tramp-compat
parse-time iso8601 ls-lisp ps-print ps-print-loaddefs ps-def lpr picture
nm dbus xml magit-submodule magit-obsolete magit-blame magit-stash
magit-reflog magit-bisect magit-push magit-pull magit-fetch magit-clone
magit-remote magit-commit magit-sequence magit-notes magit-worktree
magit-tag magit-merge magit-branch magit-reset magit-files magit-refs
magit-status magit magit-repos magit-apply magit-wip magit-log
which-func imenu magit-diff smerge-mode diff diff-mode easy-mmode
git-commit log-edit pcvs-util add-log magit-core magit-autorevert
autorevert filenotify magit-margin magit-transient magit-process
with-editor shell pcomplete server magit-mode transient comp comp-cstr
warnings rx cl-extra edmacro kmacro help-mode format-spec magit-git
magit-section magit-utils crm dash nnimap nnmail gnus-int gnus-range
mail-source message yank-media rfc822 mml mml-sec epa mm-decode
mm-bodies mm-encode mailabbrev gmm-utils mailheader utf7 netrc nnoo gnus
wid-edit nnheader gnus-util time-date mail-utils gnus-o365-oauth2 oauth2
url-http url-auth mail-parse rfc2231 rfc2047 rfc2045 mm-util ietf-drums
mail-prsvr url-gw nsm rmc puny plstore generated generic-x epg rfc6068
epg-config ediff ediff-merg ediff-mult ediff-wind ediff-diff ediff-help
ediff-init ediff-util dired-x dired dired-loaddefs compile
text-property-search comint ring ansi-color cc-mode cc-fonts cc-guess
cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs derived
debian-el info package browse-url url url-proxy url-privacy url-expand
url-methods url-history url-cookie url-domsuf url-util mailcap
url-handlers url-parse auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache json map url-vars seq gv subr-x byte-opt
bytecomp byte-compile cconv cl-loaddefs cl-lib iso-transl tooltip eldoc
paren electric uniquify ediff-hook vc-hooks lisp-float-type elisp-mode
mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite emoji-zwj charscript charprop case-table
epa-hook jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice
button loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads dbusbind inotify lcms2
dynamic-setting system-font-setting font-render-setting cairo x-toolkit
xinput2 x multi-tty make-network-process native-compile emacs)

-- 
Michael Welsh Duggan
(md5i@md5i.com)

bug#53352: 29.0.50; IMAP INBOX not refreshing using `gnus-group-get-new-news'

[Lars Ingebrigtsen]

Michael Welsh Duggan <mwd@md5i.com> writes:

> I have a gnus installation where the select method is as such:
>
>     ;;; Select Method
>     (setq gnus-select-method
>           '(nnimap "localimap"
>                     (nnimap-inbox "INBOX")
>                     (nnimap-address "xxxxxxxx.xxx")))

If you're not using Gnus to split the mail, wouldn't it make more sense
to leave nnimap-inbox as nil?

> Mail splitting is currently being handled by sieve, not by Gnus.
> However, when sieve leaves a message in INBOX rather than sorting it to
> another folder, it does not appear in INBOX using `g'
> (gnus-group-get-new-news).  Any mail that stays there does show up if I
> use `M-g' (gnus-topic-get-new-news-this-topic) on the group.  This only
> happens for the INBOX folder.  I don't believe INBOX should behave
> differently from other folders in this regard.

I don't see any special handling of the folder called INBOX in nnimap.
Do you have the INBOX group on a too-high level?  You said you're using
a group activation level of 1, but not what the level of the INBOX group
was.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#53352: 29.0.50; IMAP INBOX not refreshing using `gnus-group-get-new-news'

[Michael Welsh Duggan]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Michael Welsh Duggan <mwd@md5i.com> writes:
>
>> I have a gnus installation where the select method is as such:
>>
>>     ;;; Select Method
>>     (setq gnus-select-method
>>           '(nnimap "localimap"
>>                     (nnimap-inbox "INBOX")
>>                     (nnimap-address "xxxxxxxx.xxx")))
>
> If you're not using Gnus to split the mail, wouldn't it make more sense
> to leave nnimap-inbox as nil?

Likely.  This is a leftover from when I was using Gnus to do splitting
instead of sieve.  However, removing the `nnimap-inbox' clause and
restarting Gnus does not cause the unwanted behavior to change.

>> Mail splitting is currently being handled by sieve, not by Gnus.
>> However, when sieve leaves a message in INBOX rather than sorting it to
>> another folder, it does not appear in INBOX using `g'
>> (gnus-group-get-new-news).  Any mail that stays there does show up if I
>> use `M-g' (gnus-topic-get-new-news-this-topic) on the group.  This only
>> happens for the INBOX folder.  I don't believe INBOX should behave
>> differently from other folders in this regard.
>
> I don't see any special handling of the folder called INBOX in nnimap.
> Do you have the INBOX group on a too-high level?  You said you're using
> a group activation level of 1, but not what the level of the INBOX group
> was.

INBOX is at level 1.

-- 
Michael Welsh Duggan
(md5i@md5i.com)

bug#53352: 29.0.50; IMAP INBOX not refreshing using `gnus-group-get-new-news'

[Lars Ingebrigtsen]

Michael Welsh Duggan <mwd@md5i.com> writes:

>> I don't see any special handling of the folder called INBOX in nnimap.
>> Do you have the INBOX group on a too-high level?  You said you're using
>> a group activation level of 1, but not what the level of the INBOX group
>> was.
>
> INBOX is at level 1.

Could you try to (setq nnimap-record-commands t) and then `g' in Gnus
and then post the contents of the *imap log* buffer?  That'll tell us
which commands we're sending to the IMAP server.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#53352: 29.0.50; IMAP INBOX not refreshing using `gnus-group-get-new-news'

[Michael Welsh Duggan]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Michael Welsh Duggan <mwd@md5i.com> writes:
>
>>> I don't see any special handling of the folder called INBOX in nnimap.
>>> Do you have the INBOX group on a too-high level?  You said you're using
>>> a group activation level of 1, but not what the level of the INBOX group
>>> was.
>>
>> INBOX is at level 1.
>
> Could you try to (setq nnimap-record-commands t) and then `g' in Gnus
> and then post the contents of the *imap log* buffer?  That'll tell us
> which commands we're sending to the IMAP server.

I'm not going to send the whole list unless you believe it's necessary.
The log (after `g') consists of 122 lines as follows:

11:42:43 [md5i.com] 91340 EXAMINE "mail.misc" (QRESYNC (1044651295 71865))
11:42:43 [md5i.com] 91341 EXAMINE "mail.junk" (QRESYNC (1162282036 126251))

(I removed the ^M at the end of log lines for this email.)  This list
does not contain "INBOX".  If you actually need the full list, I can
send it, but the line above is representative of them all.

`M-g' on the INBOX group elicits the following:

11:50:58 [md5i.com] 91840 EXAMINE "INBOX" (QRESYNC (1374073954 6508))
11:50:58 [md5i.com] 91841 SELECT "INBOX"
11:50:58 [md5i.com] 91842 UID FETCH 1:* FLAGS
11:50:58 [md5i.com] 91843 EXPUNGE

Entering the server buffer and typing `RET' on the IMAP server yields:

11:51:44 [md5i.com] 91844 LIST "" "*"
11:51:44 [md5i.com] 91845 EXAMINE "Archive"
... [122 elided lines]
11:51:44 [md5i.com] 91968 EXAMINE "INBOX"

And, in case it's important, yes, the last entry is "INBOX" in that
last log snippet.

-- 
Michael Welsh Duggan
(md5i@md5i.com)

bug#53352: 29.0.50; IMAP INBOX not refreshing using `gnus-group-get-new-news'

[Lars Ingebrigtsen]

Michael Welsh Duggan <mwd@md5i.com> writes:

> I'm not going to send the whole list unless you believe it's necessary.
> The log (after `g') consists of 122 lines as follows:
>
> 11:42:43 [md5i.com] 91340 EXAMINE "mail.misc" (QRESYNC (1044651295 71865))
> 11:42:43 [md5i.com] 91341 EXAMINE "mail.junk" (QRESYNC (1162282036 126251))
>
> (I removed the ^M at the end of log lines for this email.)  This list
> does not contain "INBOX".  If you actually need the full list, I can
> send it, but the line above is representative of them all.

When I try the same, I do get INBOX in the list of EXAMINES:

10:14:57 [quimby.gnus.org] 12526 EXAMINE "Sent" (QRESYNC (1457952636 8))
10:14:57 [quimby.gnus.org] 12527 EXAMINE "INBOX" (QRESYNC (1164213559 1225378))

So it sounds like somehow your Gnus is deciding not to activate the
INBOX group (because of some setting or other, I'd guess).  I'm not sure
how you could debug that -- perhaps debug-on-entry on
nnimap-retrieve-group-data-early and see where the list of `infos' is
coming from, and see whether it includes INBOX and whether it's being
filtered out?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#53352: 29.0.50; IMAP INBOX not refreshing using `gnus-group-get-new-news'

[Michael Welsh Duggan]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Michael Welsh Duggan <mwd@md5i.com> writes:
>
>> I'm not going to send the whole list unless you believe it's necessary.
>> The log (after `g') consists of 122 lines as follows:
>>
>> 11:42:43 [md5i.com] 91340 EXAMINE "mail.misc" (QRESYNC (1044651295 71865))
>> 11:42:43 [md5i.com] 91341 EXAMINE "mail.junk" (QRESYNC (1162282036 126251))
>>
>> (I removed the ^M at the end of log lines for this email.)  This list
>> does not contain "INBOX".  If you actually need the full list, I can
>> send it, but the line above is representative of them all.
>
> When I try the same, I do get INBOX in the list of EXAMINES:
>
> 10:14:57 [quimby.gnus.org] 12526 EXAMINE "Sent" (QRESYNC (1457952636 8))
> 10:14:57 [quimby.gnus.org] 12527 EXAMINE "INBOX" (QRESYNC (1164213559 1225378))
>
> So it sounds like somehow your Gnus is deciding not to activate the
> INBOX group (because of some setting or other, I'd guess).  I'm not sure
> how you could debug that -- perhaps debug-on-entry on
> nnimap-retrieve-group-data-early and see where the list of `infos' is
> coming from, and see whether it includes INBOX and whether it's being
> filtered out?

This appears to be happening in `gnus-get-unread-articles' in the
initial `let*'.  "INBOX" is the car of `gnus-newsrc-alist', and `newsrc'
is explicitly set to the cdr of that list.

-- 
Michael Welsh Duggan
(md5i@md5i.com)

bug#53352: 29.0.50; IMAP INBOX not refreshing using `gnus-group-get-new-news'

[Lars Ingebrigtsen]

Michael Welsh Duggan <mwd@md5i.com> writes:

> This appears to be happening in `gnus-get-unread-articles' in the
> initial `let*'.  "INBOX" is the car of `gnus-newsrc-alist', and `newsrc'
> is explicitly set to the cdr of that list.

Oh!  This must be a leftover from when `gnus-newsrc-alist' always had a
dummy first entry, but that was changed a while back.  I think?

Eric, what's the status of the dummy.group that that used to be the
first element of gnus-newsrc-alist, but doesn't seem to be any more?

There's still a whole bunch of stuff in Gnus that skips the first
element because of that:

./lisp/gnus/gnus-sieve.el211:  (let* ((newsrc (cdr gnus-newsrc-alist))
./lisp/gnus/gnus-kill.el653:    (setq newsrc (cdr gnus-newsrc-alist))
./lisp/gnus/gnus-topic.el896:			    (cdr gnus-newsrc-alist)))
./lisp/gnus/gnus-cus.el1083:                                    (cdr gnus-newsrc-alist)))
./lisp/gnus/gnus-cus.el1086:                                      (cdr gnus-newsrc-alist))))))))
./lisp/gnus/gnus-group.el1942:  (let ((alist (cdr gnus-newsrc-alist))

Do I misremember, or did you change this?  At least there's no
dummy.group as the car of my (and Michaels) gnus-newsrc-alist-s

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#53352: 29.0.50; IMAP INBOX not refreshing using `gnus-group-get-new-news'

[Andreas Schwab]

On Jan 24 2022, Michael Welsh Duggan wrote:

> This appears to be happening in `gnus-get-unread-articles' in the
> initial `let*'.  "INBOX" is the car of `gnus-newsrc-alist', and `newsrc'
> is explicitly set to the cdr of that list.

The first element of gnus-newsrc-alist is supposed to be the
dummy.group, but that doesn't appear to be the case any more.  Can this
be related to commit a09306d53b?

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#53352: 29.0.50; IMAP INBOX not refreshing using `gnus-group-get-new-news'

[Eric Abrahamsen]

On 01/24/22 17:34 PM, Andreas Schwab wrote:
> On Jan 24 2022, Michael Welsh Duggan wrote:
>
>> This appears to be happening in `gnus-get-unread-articles' in the
>> initial `let*'.  "INBOX" is the car of `gnus-newsrc-alist', and `newsrc'
>> is explicitly set to the cdr of that list.
>
> The first element of gnus-newsrc-alist is supposed to be the
> dummy.group, but that doesn't appear to be the case any more.  Can this
> be related to commit a09306d53b?

Yup, looks like that's the problem. After the first time you save Gnus,
`gnus-newsrc-alist' has been redefined to exclude "dummy.group". It
should be easy to fix -- I'll move the sorting and exclusion into the
`let', which will preserve the variable's original value.

I suppose the Emacs 28 ship has sailed? It would be nice to include this
fix in the release.

Eric

bug#53352: 29.0.50; IMAP INBOX not refreshing using `gnus-group-get-new-news'

[Michael Welsh Duggan]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> On 01/24/22 17:34 PM, Andreas Schwab wrote:
>> On Jan 24 2022, Michael Welsh Duggan wrote:
>>
>>> This appears to be happening in `gnus-get-unread-articles' in the
>>> initial `let*'.  "INBOX" is the car of `gnus-newsrc-alist', and `newsrc'
>>> is explicitly set to the cdr of that list.
>>
>> The first element of gnus-newsrc-alist is supposed to be the
>> dummy.group, but that doesn't appear to be the case any more.  Can this
>> be related to commit a09306d53b?
>
> Yup, looks like that's the problem. After the first time you save Gnus,
> `gnus-newsrc-alist' has been redefined to exclude "dummy.group". It
> should be easy to fix -- I'll move the sorting and exclusion into the
> `let', which will preserve the variable's original value.

To be clear, would this mean I need to add "dummy.group" to my
`gnus-newsrc-alist' after you make this change, or will the change
obviate the need for the dummy entry?  If the former, would you please
verify the form I need to prepend to the alist?  Is it just the
"dummy.group" string?

-- 
Michael Welsh Duggan
(md5i@md5i.com)

bug#53352: 29.0.50; IMAP INBOX not refreshing using `gnus-group-get-new-news'

[Lars Ingebrigtsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> I suppose the Emacs 28 ship has sailed? It would be nice to include this
> fix in the release.

It's a regression from 27.2, so including a fix in Emacs 28 would be
appropriate (if it's simple enough).  But we can run it on master for a
while before backporting.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#53352: 29.0.50; IMAP INBOX not refreshing using `gnus-group-get-new-news'

[Eric Abrahamsen]

On 01/24/22 13:06 PM, Michael Welsh Duggan wrote:
> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> On 01/24/22 17:34 PM, Andreas Schwab wrote:
>>> On Jan 24 2022, Michael Welsh Duggan wrote:
>>>
>>>> This appears to be happening in `gnus-get-unread-articles' in the
>>>> initial `let*'.  "INBOX" is the car of `gnus-newsrc-alist', and `newsrc'
>>>> is explicitly set to the cdr of that list.
>>>
>>> The first element of gnus-newsrc-alist is supposed to be the
>>> dummy.group, but that doesn't appear to be the case any more.  Can this
>>> be related to commit a09306d53b?
>>
>> Yup, looks like that's the problem. After the first time you save Gnus,
>> `gnus-newsrc-alist' has been redefined to exclude "dummy.group". It
>> should be easy to fix -- I'll move the sorting and exclusion into the
>> `let', which will preserve the variable's original value.
>
> To be clear, would this mean I need to add "dummy.group" to my
> `gnus-newsrc-alist' after you make this change, or will the change
> obviate the need for the dummy entry?  If the former, would you please
> verify the form I need to prepend to the alist?  Is it just the
> "dummy.group" string?

You won't have to do anything. With the code as it is, the dummy group
is present at Gnus startup, but gets deleted on first save. I'll put in
a fix that makes sure it doesn't get deleted. That should take care of
things.

There's no longer any need for a dummy group, and in the long run it
would be good to get rid of it. But I think in short term we still think
there's a significant amount of out-of-tree code that might be expecting
a dummy group.

Eric

bug#53352: 29.0.50; IMAP INBOX not refreshing using `gnus-group-get-new-news'

[Eric Abrahamsen]

[-- Attachment #1: Type: text/plain, Size: 500 bytes --]


On 01/24/22 19:31 PM, Lars Ingebrigtsen wrote:
> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> I suppose the Emacs 28 ship has sailed? It would be nice to include this
>> fix in the release.
>
> It's a regression from 27.2, so including a fix in Emacs 28 would be
> appropriate (if it's simple enough).  But we can run it on master for a
> while before backporting.

Okay, good. I'm attaching the patch I'm using right now -- this appears
to fix it. I'll give it another day or two.

Eric


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: preserve-gnus-newsrc-alist.diff --]
[-- Type: text/x-patch, Size: 2200 bytes --]

diff --git a/lisp/gnus/gnus-start.el b/lisp/gnus/gnus-start.el
index 2cf11fb12f..dd9c277805 100644
--- a/lisp/gnus/gnus-start.el
+++ b/lisp/gnus/gnus-start.el
@@ -2867,12 +2867,6 @@ gnus-gnus-to-quick-newsrc-format
       (princ "(setq gnus-newsrc-file-version ")
       (princ (gnus-prin1-to-string gnus-version))
       (princ ")\n"))
-    ;; Sort `gnus-newsrc-alist' according to order in
-    ;; `gnus-group-list'.
-    (setq gnus-newsrc-alist
-	  (mapcar (lambda (g)
-		    (nth 1 (gethash g gnus-newsrc-hashtb)))
-		  (delete "dummy.group" gnus-group-list)))
     (let* ((print-quoted t)
            (print-escape-multibyte nil)
            (print-escape-nonascii t)
@@ -2891,17 +2885,20 @@ gnus-gnus-to-quick-newsrc-format
 		  ;; Remove the `gnus-killed-list' from the list of variables
 		  ;; to be saved, if required.
 		  (delq 'gnus-killed-list (copy-sequence gnus-variable-list)))))
-	   ;; Encode group names in `gnus-newsrc-alist' and
-	   ;; `gnus-topic-alist' in order to keep newsrc.eld files
-	   ;; compatible with older versions of Gnus.  At some point,
-	   ;; if/when a new version of Gnus is released, stop doing
-	   ;; this and move the corresponding decode in
-	   ;; `gnus-read-newsrc-el-file' into a conversion routine.
+           ;; Sort `gnus-newsrc-alist' according to order in
+           ;; `gnus-group-list'.  Encode group names in
+           ;; `gnus-newsrc-alist' and `gnus-topic-alist' in order to
+           ;; keep newsrc.eld files compatible with older versions of
+           ;; Gnus.  At some point, if/when a new version of Gnus is
+           ;; released, stop doing this and move the corresponding
+           ;; decode in `gnus-read-newsrc-el-file' into a conversion
+           ;; routine.
 	   (gnus-newsrc-alist
-	    (mapcar (lambda (info)
-		      (cons (encode-coding-string (car info) 'utf-8-emacs)
-			    (cdr info)))
-		    gnus-newsrc-alist))
+	    (mapcar (lambda (group)
+		      (cons (encode-coding-string group 'utf-8-emacs)
+			    (cdadr (gethash group
+			    gnus-newsrc-hashtb))))
+		    (remove "dummy.group" gnus-group-list)))
 	   (gnus-topic-alist
 	    (when (memq 'gnus-topic-alist variables)
 	     (mapcar (lambda (elt)

bug#53352: 29.0.50; IMAP INBOX not refreshing using `gnus-group-get-new-news'

[Michael Welsh Duggan]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> On 01/24/22 19:31 PM, Lars Ingebrigtsen wrote:
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>>> I suppose the Emacs 28 ship has sailed? It would be nice to include this
>>> fix in the release.
>>
>> It's a regression from 27.2, so including a fix in Emacs 28 would be
>> appropriate (if it's simple enough).  But we can run it on master for a
>> while before backporting.
>
> Okay, good. I'm attaching the patch I'm using right now -- this appears
> to fix it. I'll give it another day or two.

I can confirm that this solves the problem for me, at least for some
simple local testing.  I'll run with this patch applied for a few days
as well.

-- 
Michael Welsh Duggan
(md5i@md5i.com)

bug#53352: 29.0.50; IMAP INBOX not refreshing using `gnus-group-get-new-news'

[Eric Abrahamsen]

Michael Welsh Duggan <mwd@md5i.com> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> On 01/24/22 19:31 PM, Lars Ingebrigtsen wrote:
>>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>>
>>>> I suppose the Emacs 28 ship has sailed? It would be nice to include this
>>>> fix in the release.
>>>
>>> It's a regression from 27.2, so including a fix in Emacs 28 would be
>>> appropriate (if it's simple enough).  But we can run it on master for a
>>> while before backporting.
>>
>> Okay, good. I'm attaching the patch I'm using right now -- this appears
>> to fix it. I'll give it another day or two.
>
> I can confirm that this solves the problem for me, at least for some
> simple local testing.  I'll run with this patch applied for a few days
> as well.

Great, thank you! I'll aim to push on say Thursday, then we can let it
simmer on master for a bit, then backport to 28.

bug#53352: 29.0.50; IMAP INBOX not refreshing using `gnus-group-get-new-news'

[Michael Welsh Duggan]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Michael Welsh Duggan <mwd@md5i.com> writes:
>
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>>> On 01/24/22 19:31 PM, Lars Ingebrigtsen wrote:
>>>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>>>
>>>>> I suppose the Emacs 28 ship has sailed? It would be nice to include this
>>>>> fix in the release.
>>>>
>>>> It's a regression from 27.2, so including a fix in Emacs 28 would be
>>>> appropriate (if it's simple enough).  But we can run it on master for a
>>>> while before backporting.
>>>
>>> Okay, good. I'm attaching the patch I'm using right now -- this appears
>>> to fix it. I'll give it another day or two.
>>
>> I can confirm that this solves the problem for me, at least for some
>> simple local testing.  I'll run with this patch applied for a few days
>> as well.
>
> Great, thank you! I'll aim to push on say Thursday, then we can let it
> simmer on master for a bit, then backport to 28.

Just a ping to make sure this didn't get forgotten.

-- 
Michael Welsh Duggan
(md5i@md5i.com)

bug#53352: 29.0.50; IMAP INBOX not refreshing using `gnus-group-get-new-news'

[Eric Abrahamsen]

On 01/31/22 09:32 AM, Michael Welsh Duggan wrote:
> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Michael Welsh Duggan <mwd@md5i.com> writes:
>>
>>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>>
>>>> On 01/24/22 19:31 PM, Lars Ingebrigtsen wrote:
>>>>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>>>>
>>>>>> I suppose the Emacs 28 ship has sailed? It would be nice to include this
>>>>>> fix in the release.
>>>>>
>>>>> It's a regression from 27.2, so including a fix in Emacs 28 would be
>>>>> appropriate (if it's simple enough).  But we can run it on master for a
>>>>> while before backporting.
>>>>
>>>> Okay, good. I'm attaching the patch I'm using right now -- this appears
>>>> to fix it. I'll give it another day or two.
>>>
>>> I can confirm that this solves the problem for me, at least for some
>>> simple local testing.  I'll run with this patch applied for a few days
>>> as well.
>>
>> Great, thank you! I'll aim to push on say Thursday, then we can let it
>> simmer on master for a bit, then backport to 28.
>
> Just a ping to make sure this didn't get forgotten.

Not forgotten! But it was languishing in my local rebase pile, just
pushed.

I'm going to close this now, but please reopen if anything seems not to
be working as it should.

Eric

bug#53352: 29.0.50; IMAP INBOX not refreshing using `gnus-group-get-new-news'

[Andreas Schwab]

On Jan 24 2022, Eric Abrahamsen wrote:

> diff --git a/lisp/gnus/gnus-start.el b/lisp/gnus/gnus-start.el
> index 2cf11fb12f..dd9c277805 100644
> --- a/lisp/gnus/gnus-start.el
> +++ b/lisp/gnus/gnus-start.el
> @@ -2867,12 +2867,6 @@ gnus-gnus-to-quick-newsrc-format
>        (princ "(setq gnus-newsrc-file-version ")
>        (princ (gnus-prin1-to-string gnus-version))
>        (princ ")\n"))
> -    ;; Sort `gnus-newsrc-alist' according to order in
> -    ;; `gnus-group-list'.
> -    (setq gnus-newsrc-alist
> -	  (mapcar (lambda (g)
> -		    (nth 1 (gethash g gnus-newsrc-hashtb)))
> -		  (delete "dummy.group" gnus-group-list)))

But the sorting should still be permanent, shoudn't it?

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#53352: 29.0.50; IMAP INBOX not refreshing using `gnus-group-get-new-news'

[Eric Abrahamsen]

On 02/07/22 11:49 AM, Andreas Schwab wrote:
> On Jan 24 2022, Eric Abrahamsen wrote:
>
>> diff --git a/lisp/gnus/gnus-start.el b/lisp/gnus/gnus-start.el
>> index 2cf11fb12f..dd9c277805 100644
>> --- a/lisp/gnus/gnus-start.el
>> +++ b/lisp/gnus/gnus-start.el
>> @@ -2867,12 +2867,6 @@ gnus-gnus-to-quick-newsrc-format
>>        (princ "(setq gnus-newsrc-file-version ")
>>        (princ (gnus-prin1-to-string gnus-version))
>>        (princ ")\n"))
>> -    ;; Sort `gnus-newsrc-alist' according to order in
>> -    ;; `gnus-group-list'.
>> -    (setq gnus-newsrc-alist
>> -	  (mapcar (lambda (g)
>> -		    (nth 1 (gethash g gnus-newsrc-hashtb)))
>> -		  (delete "dummy.group" gnus-group-list)))
>
> But the sorting should still be permanent, shoudn't it?

It really doesn't matter, at this point. `gnus-group-list' is the source
of authority on sorting (if the user isn't using topic mode), and
`gnus-newsrc-hashtb' is more and more just used for persisting to disk.
We want to save the sort order on shutdown, but during a Gnus session it
doesn't matter.

bug#53352: 29.0.50; IMAP INBOX not refreshing using `gnus-group-get-new-news'

[Andreas Schwab]

On Feb 07 2022, Eric Abrahamsen wrote:

> On 02/07/22 11:49 AM, Andreas Schwab wrote:
>> On Jan 24 2022, Eric Abrahamsen wrote:
>>
>>> diff --git a/lisp/gnus/gnus-start.el b/lisp/gnus/gnus-start.el
>>> index 2cf11fb12f..dd9c277805 100644
>>> --- a/lisp/gnus/gnus-start.el
>>> +++ b/lisp/gnus/gnus-start.el
>>> @@ -2867,12 +2867,6 @@ gnus-gnus-to-quick-newsrc-format
>>>        (princ "(setq gnus-newsrc-file-version ")
>>>        (princ (gnus-prin1-to-string gnus-version))
>>>        (princ ")\n"))
>>> -    ;; Sort `gnus-newsrc-alist' according to order in
>>> -    ;; `gnus-group-list'.
>>> -    (setq gnus-newsrc-alist
>>> -	  (mapcar (lambda (g)
>>> -		    (nth 1 (gethash g gnus-newsrc-hashtb)))
>>> -		  (delete "dummy.group" gnus-group-list)))
>>
>> But the sorting should still be permanent, shoudn't it?
>
> It really doesn't matter, at this point.

It doesn't?  I get erroneous behaviour without this.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#53352: 29.0.50; IMAP INBOX not refreshing using `gnus-group-get-new-news'

[Eric Abrahamsen]

On 02/07/22 18:42 PM, Andreas Schwab wrote:
> On Feb 07 2022, Eric Abrahamsen wrote:
>
>> On 02/07/22 11:49 AM, Andreas Schwab wrote:
>>> On Jan 24 2022, Eric Abrahamsen wrote:
>>>
>>>> diff --git a/lisp/gnus/gnus-start.el b/lisp/gnus/gnus-start.el
>>>> index 2cf11fb12f..dd9c277805 100644
>>>> --- a/lisp/gnus/gnus-start.el
>>>> +++ b/lisp/gnus/gnus-start.el
>>>> @@ -2867,12 +2867,6 @@ gnus-gnus-to-quick-newsrc-format
>>>>        (princ "(setq gnus-newsrc-file-version ")
>>>>        (princ (gnus-prin1-to-string gnus-version))
>>>>        (princ ")\n"))
>>>> -    ;; Sort `gnus-newsrc-alist' according to order in
>>>> -    ;; `gnus-group-list'.
>>>> -    (setq gnus-newsrc-alist
>>>> -	  (mapcar (lambda (g)
>>>> -		    (nth 1 (gethash g gnus-newsrc-hashtb)))
>>>> -		  (delete "dummy.group" gnus-group-list)))
>>>
>>> But the sorting should still be permanent, shoudn't it?
>>
>> It really doesn't matter, at this point.
>
> It doesn't?  I get erroneous behaviour without this.

It shouldn't -- if you're not using topic mode, the group sorting is
drawn from `gnus-group-list'. And when you save->shutdown->restart Gnus,
`gnus-group-list' sorts `gnus-newsrc-alist', which gets written to file,
and on restart it is used to reconstruct the sort order for
`gnus-newsrc-alist'.

Can you tell me what the erroneous behaviour is, and also what "this"
refers to?

bug#53352: 29.0.50; IMAP INBOX not refreshing using `gnus-group-get-new-news'

[Andreas Schwab]

On Feb 07 2022, Eric Abrahamsen wrote:

> Can you tell me what the erroneous behaviour is,

Group rearrangements in *Group* are forgotten.

> and also what "this" refers to?

The hunk reverted with s/delete/remove/.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#53352: 29.0.50; IMAP INBOX not refreshing using `gnus-group-get-new-news'

[Eric Abrahamsen]

On 02/07/22 22:34 PM, Andreas Schwab wrote:
> On Feb 07 2022, Eric Abrahamsen wrote:
>
>> Can you tell me what the erroneous behaviour is,
>
> Group rearrangements in *Group* are forgotten.

`gnus-group-prepare-flat' (which I have to assume you're running) draws
its ordering from `gnus-group-list'. I'll try to reproduce your error.

bug#53352: 29.0.50; IMAP INBOX not refreshing using `gnus-group-get-new-news'

[Eric Abrahamsen]

On 02/07/22 22:34 PM, Andreas Schwab wrote:
> On Feb 07 2022, Eric Abrahamsen wrote:
>
>> Can you tell me what the erroneous behaviour is,
>
> Group rearrangements in *Group* are forgotten.

So far I haven't been able to reproduce this:

1. Turn off topic-mode, if necessary.
2. Kill a group, and yank it somewhere else.
3. Re-list groups with "l": sort order doesn't change.
4. Save Gnus, re-list with "l": sort order doesn't change.
5. Shut Gnus down, restart: sort order is preserved.

And I tried the above using the sort commands instead of killing and
yanking groups.

Can you give me a recipe to produce the error?

bug#53352: 29.0.50; IMAP INBOX not refreshing using `gnus-group-get-new-news'

[Andreas Schwab]

Try calling gnus-make-hashtable-from-newsrc-alist.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#53352: 29.0.50; IMAP INBOX not refreshing using `gnus-group-get-new-news'

[Eric Abrahamsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

> Try calling gnus-make-hashtable-from-newsrc-alist.

Nope, still can't guess what you're talking about.