bug#50450: 28.0.50; nnmaildir cannot find article file

bug#50450: 28.0.50; nnmaildir cannot find article file

[Philip Kaludercic]

From time to time I get error messages like

nnmaildir--article-set-flags: Couldn’t find article file /home/philip/Mail/Pers/Sent/cur/1631002186.M670797P704685Q0.icterid:2,S

when trying to send a message. From what I understand, the issue is that
my IMAP mail client isync (https://isync.sourceforge.io/) changes the
file name while I am writing, and changes 

/home/philip/Mail/Pers/Sent/cur/1631002186.M670797P704685Q0.icterid:2,S

to

/home/philip/Mail/Pers/Sent/cur/1631002186.M670797P704685Q0.icterid,U=711:2,S

where the U=... is apparently a maildir attribute indicating a UID
(I cannot find any good documentation on this).

As a result, an error is thrown right after the message is sent, and the
buffer is not cleaned up. Instead I see the generated flags, and have to
manually delete the buffer.

In GNU Emacs 28.0.50 (build 2, x86_64-pc-linux-gnu, X toolkit, cairo version 1.16.0, Xaw scroll bars)
 of 2021-09-06 built on icterid
Repository revision: d4ea153d8feecb55c2d9b1779574d0bb5417d120
Repository branch: feature/rcirc-update
Windowing system distributor 'The X.Org Foundation', version 11.0.12011000
System Description: Debian GNU/Linux 11 (bullseye)

Configured using:
 'configure LDFLAGS=-flto 'CFLAGS=-O2 -march=native -mtune=native -pipe'
 --with-native-compilation PKG_CONFIG_PATH='

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES NATIVE_COMP
NOTIFY INOTIFY PDUMPER PNG SECCOMP SOUND THREADS TIFF
TOOLKIT_SCROLL_BARS X11 XDBE XIM XPM LUCID ZLIB

Important settings:
  value of $EMACSLOADPATH: 
  value of $LC_ALL: C
  value of $LANG: en_US.UTF-8
  locale-coding-system: nil

Major mode: ELisp/l

Minor modes in effect:
  global-git-commit-mode: t
  magit-auto-revert-mode: t
  auto-revert-mode: t
  shell-dirtrack-mode: t
  bug-reference-prog-mode: t
  paredit-mode: t
  company-mode: t
  flymake-mode: t
  flyspell-mode: t
  outline-minor-mode: t
  rcirc-track-minor-mode: t
  display-time-mode: t
  winner-mode: t
  windmove-mode: t
  electric-pair-mode: t
  recentf-mode: t
  save-place-mode: t
  savehist-mode: t
  show-paren-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tab-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  temp-buffer-resize-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t

Load-path shadows:
/home/philip/.config/emacs/elpa/transient-0.3.6/transient hides /home/philip/Code/src/emacs/lisp/transient
/home/philip/.config/emacs/elpa/xref-1.2.2/xref hides /home/philip/Code/src/emacs/lisp/progmodes/xref
/home/philip/.config/emacs/elpa/project-0.6.1/project hides /home/philip/Code/src/emacs/lisp/progmodes/project
~/.config/emacs/site-lisp/autoload hides /home/philip/Code/src/emacs/lisp/emacs-lisp/autoload

Features:
(shadow emacsbug ffap rng-xsd xsd-regexp rng-cmpct rng-nxml rng-valid
rng-loc rng-uri rng-parse nxml-parse rng-match rng-dt rng-util rng-pttrn
nxml-ns nxml-mode nxml-outln nxml-rap sgml-mode facemenu nxml-util
nxml-enc xmltok make-mode dictionary dictionary-connection mailalias
bbdb-pgp shell-command+ tar-mode arc-mode archive-mode finder-inf
reposition ibuf-ext ibuffer ibuffer-loaddefs flymake-cc macrostep-c
cmacexp macrostep cc-mode cc-fonts cc-guess cc-menus cc-cmds cc-styles
cc-align cc-engine cc-vars cc-defs debug vc-fossil vc-mtn vc-hg vc-bzr
vc-src vc-sccs vc-svn vc-cvs vc-rcs edebug backtrace modus-vivendi-theme
nndoc url-http url-gw url-cache url-auth eieio-opt speedbar ezimage
dframe shortdoc help-at-pt sort smiley gnus-cite flow-fill mm-archive
gnus-bcklg qp gnus-async gnus-ml git-rebase magit-extras goto-addr
mule-util face-remap magit-submodule magit-obsolete magit-blame
magit-stash magit-reflog magit-bisect magit-push magit-pull magit-fetch
magit-clone magit-remote magit-commit magit-sequence magit-notes
magit-worktree magit-tag magit-merge magit-branch magit-reset
magit-files magit-refs magit-status magit magit-repos magit-apply
magit-wip magit-log which-func imenu magit-diff git-commit magit-core
magit-autorevert autorevert filenotify magit-margin magit-transient
magit-process with-editor log-edit add-log term ehelp eshell esh-cmd
esh-ext esh-opt esh-proc esh-io esh-arg esh-module esh-groups esh-util
shell pcomplete server magit-mode transient format-spec magit-git
magit-section magit-utils dash dired-aux cus-start avy bbdb-message
mail-extr autocrypt-message autocrypt-gnus autocrypt nndraft nnmh
nnmaildir nnfolder bbdb-gnus bbdb-mua bbdb-com crm nnnil gnus-agent
gnus-srvr gnus-score score-mode nnvirtual gnus-msg gnus-art mm-uu
mml2015 mm-view mml-smime smime dig nntp gnus-cache gnus-sum shr kinsoku
svg dom gnus-group gnus-undo gnus-start gnus-dbus dbus xml gnus-cloud
nnimap nnmail mail-source utf7 netrc nnoo gnus-spec gnus-int gnus-range
message rfc822 mml mml-sec mm-decode mm-bodies mm-encode mail-parse
rfc2231 mailabbrev gmm-utils mailheader gnus-win cl-print apropos pp
descr-text pulse color char-fold misearch multi-isearch tabify
time-stamp smerge-mode whitespace vc-git diff-mode bug-reference
find-func vc-backup log-view pcvs-util vc vc-dispatcher diff disp-table
gnutls network-stream puny nsm rmc epa-file epa epg epg-config paredit
company-oddmuse company-keywords company-etags etags fileloop generator
xref project company-gtags company-dabbrev-code company-dabbrev
company-files company-capf company-cmake company-xcode company-clang
company-semantic company-eclim company-template company-bbdb company
checkdoc flymake-proc flymake thingatpt flyspell ispell noutline outline
easy-mmode gnus-dired dired-x dired dired-loaddefs rcirc parse-time
iso8601 time bbdb derived bbdb-site timezone sendmail gnus nnheader
gnus-util rmail rmail-loaddefs rfc2047 rfc2045 ietf-drums time-date
mail-utils mm-util mail-prsvr hippie-exp winner windmove elec-pair
recentf tree-widget wid-edit saveplace savehist paren
modus-operandi-theme modus-themes pcase edmacro kmacro holidays
hol-loaddefs cal-menu calendar cal-loaddefs cus-load setup load compile
text-property-search comint ansi-color autoload lisp-mnt tex-site
geiser-impl help-fns radix-tree geiser-custom geiser-base ring
slime-autoloads info package browse-url url url-proxy url-privacy
url-expand url-methods url-history url-cookie url-domsuf url-util
mailcap url-handlers url-parse auth-source comp comp-cstr warnings rx
cl-seq cl-extra help-mode eieio eieio-core cl-macs eieio-loaddefs
password-cache json subr-x map url-vars seq byte-opt gv bytecomp
byte-compile cconv cl-loaddefs cl-lib iso-transl tooltip eldoc electric
uniquify ediff-hook vc-hooks lisp-float-type mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice button
loaddefs faces cus-face macroexp files window text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads dbusbind inotify
dynamic-setting system-font-setting font-render-setting cairo x-toolkit
x multi-tty make-network-process native-compile emacs)

Memory information:
((conses 16 1011811 796046)
 (symbols 48 52819 374)
 (strings 32 253617 140796)
 (string-bytes 1 8134234)
 (vectors 16 96296)
 (vector-slots 8 2662425 1290279)
 (floats 8 903 3581)
 (intervals 56 34206 18940)
 (buffers 992 82))

-- 
	Philip K.

bug#50450: 28.0.50; nnmaildir cannot find article file

[Lars Ingebrigtsen]

Philip Kaludercic <philipk@posteo.net> writes:

>From time to time I get error messages like
>
> nnmaildir--article-set-flags: Couldn’t find article file /home/philip/Mail/Pers/Sent/cur/1631002186.M670797P704685Q0.icterid:2,S
>
> when trying to send a message.

I don't use nnmaildir myself (and I'm not familiar with how maildir
works, really).  Can you propose a patch to fix this yourself?  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#50450: 28.0.50; nnmaildir cannot find article file

[Philip Kaludercic]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>>From time to time I get error messages like
>>
>> nnmaildir--article-set-flags: Couldn’t find article file /home/philip/Mail/Pers/Sent/cur/1631002186.M670797P704685Q0.icterid:2,S
>>
>> when trying to send a message.
>
> I don't use nnmaildir myself (and I'm not familiar with how maildir
> works, really).  Can you propose a patch to fix this yourself?  :-)

I would have, but I am not familiar with the inner workings for Gnus,
and as nnmaildir--article-set-flags was not documented, I was hesitant
to change how it behaves. But I'll try to see if I can fix
anything. Hopefully it is ok to come back to this thread if I have
questions.

-- 
	Philip Kaludercic

bug#50450: 28.0.50; nnmaildir cannot find article file

[Lars Ingebrigtsen]

Philip Kaludercic <philipk@posteo.net> writes:

> But I'll try to see if I can fix anything.

Great.

> Hopefully it is ok to come back to this thread if I have questions.

Yes, of course.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#50450: 28.0.50; nnmaildir cannot find article file

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 999 bytes --]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>>From time to time I get error messages like
>>
>> nnmaildir--article-set-flags: Couldn’t find article file
>> /home/philip/Mail/Pers/Sent/cur/1631002186.M670797P704685Q0.icterid:2,S
>>
>> when trying to send a message.
>
> I don't use nnmaildir myself (and I'm not familiar with how maildir
> works, really).  Can you propose a patch to fix this yourself?  :-)

I found out that the issue is not the UID, but the flags. Specifically,
when I open a message via Gnus, leave it open for a while, perhaps check
the message on my phone or some other device, and then return to Gnus,
it might have happened that the flags (included in the file-name) were
updated by mbsync.

This patch attempts to find a similar file, and renames that file
instead of the one Gnus remembered. It seems to be working for now, but
I'd wait a bit until I had more time to test it before applying it:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Handle-updates-flags-when-setting-flags.patch --]
[-- Type: text/x-diff, Size: 1401 bytes --]

From facb1848952d5c844d76844e35da708fda8307c7 Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Wed, 15 Sep 2021 10:58:50 +0200
Subject: [PATCH] Handle updates flags when setting flags

* nnmaildir.el (nnmaildir--article-set-flags): Handle updated flags
  more gracefully
---
 lisp/gnus/nnmaildir.el | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/lisp/gnus/nnmaildir.el b/lisp/gnus/nnmaildir.el
index 171f0813b3..690761a2d6 100644
--- a/lisp/gnus/nnmaildir.el
+++ b/lisp/gnus/nnmaildir.el
@@ -194,7 +194,15 @@ nnmaildir--article-set-flags
 	 (article-file (concat curdir prefix suffix))
 	 (new-name (concat curdir prefix new-suffix)))
     (unless (file-exists-p article-file)
-      (error "Couldn't find article file %s" article-file))
+      (let ((possible (file-expand-wildcards (concat curdir prefix "*"))))
+	(cond ((length= possible 1)
+	       (unless (string-match-p "\\`\\(.+\\):2,.*?\\'" (car possible))
+		 (error "Couldn't find updated article file %s" article-file))
+	       (setq article-file (car possible)))
+	      ((length> possible 1)
+	       (error "Couldn't determine exact article file %s" article-file))
+	      ((null possible)
+	       (error "Couldn't find article file %s" article-file)))))
     (rename-file article-file new-name 'replace)
     (setf (nnmaildir--art-suffix article) new-suffix)))
 
-- 
2.30.2


[-- Attachment #3: Type: text/plain, Size: 147 bytes --]


One possible issue is that nnmaildir--article-set-flags overwrites
previous flags that might have been updated elsewhere.

-- 
	Philip Kaludercic

bug#50450: 28.0.50; nnmaildir cannot find article file

[Lars Ingebrigtsen]

Philip Kaludercic <philipk@posteo.net> writes:

> This patch attempts to find a similar file, and renames that file
> instead of the one Gnus remembered. It seems to be working for now, but
> I'd wait a bit until I had more time to test it before applying it:

The patch looks good to me, so go ahead and push it after testing it
some more.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#50450: 28.0.50; nnmaildir cannot find article file

[Philip Kaludercic]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> This patch attempts to find a similar file, and renames that file
>> instead of the one Gnus remembered. It seems to be working for now, but
>> I'd wait a bit until I had more time to test it before applying it:
>
> The patch looks good to me, so go ahead and push it after testing it
> some more.

I could just confirm that the patch fixes the bug. As I haven't had any
issues up until now, I pushed the commit.

-- 
	Philip Kaludercic