bug#67890: 28.2; nnmaildir silently deletes messages upon expiration

[Eli Zaretskii <eliz@gnu.org>]

> From: Łukasz Stelmach <stlman@poczta.fm>
> Date: Tue, 19 Dec 2023 00:10:11 +0100
> 
> I believe I've found a bug leading to a data loss situation.
> 
> Steps to reproduce.
> 
> 1. Setup a maildir virtual server
> 
> 2. Create two or more groups in it.
> 
> 3. Set expire-age and expire-group parameters (G p) for one so that
>    messages would be moved to the other.
> 
> 4. Exit Gnus and Emacs
> 
> 5. Deliver a message to that group (let's call it source group) with a
>    date older than expire-age (for example using the script below)
> 
> 6. Run Gnus
> 
> 7. (Optional) M-x describe-function nnmaildir--pgname. Go to the
> function and C-u M-x eval-defun.
> 
> 8. Enter the source group.
> 
> 9. (Optional) The debugger will stop in nnmaildir--pgname. Go step by
>    step and notice the prefix (inside the let) will become an empty
>    string. This string will be also set as the prefix of the server
>    passed to the function.
> 
> 10. (Optional though recommended) Enable debugging for
>     nnmaildir-request-expire-articles.
> 
> 11. Mark the message delivered in point 5 as expirable (E)
> 
> 12. Exit the group.
> 
> 13. (Optional depending on point 10) The debugger will stop in
>     nnmaildir-request-expire-articles. Skip to the point where the pgname
>     variable is set. With debugging enabled for nnmaildir--pgname you
>     can follow the entire process and notice that pgname won't have
>     appropriate prefix and will be (most probably) equal to gname. Go on
>     to the point where the target variable is set
> 
> --8<---------------cut here---------------start------------->8---
>     target (if force nil
>              (save-excursion
>                (save-restriction
>                  (nnmaildir--param pgname 'expire-group)))))
> --8<---------------cut here---------------end--------------->8---
> 
>    At this point when nnmaildir--param is set with a group name without
>    a prefix it will return nil which will prevent the message from being
>    moved in the next few lines
> 
> --8<---------------cut here---------------start------------->8---
>              (when (and (stringp target)
>                         (not (string-equal target pgname))) ;; Move it.
> --8<---------------cut here---------------end--------------->8---
> 
>    but not from being unlinked near the end of the
>    nnmaildir-request-expire-articls function.
>    
> 14. The message gets totally removed.
> 
> The major problem seems to araise from gnus-group-prefixed-name not
> finding proper group prefix, which happens because the method field of
> nnmaildir--srv doesn't seem to be properly set in nnmaildir--prepare.
> 
> I am digging deeper and deeper but I can't find anything obvious, so I
> think this is the moment I will share me findings anyway.
> 
> As far as I can tell this scenario takes place only when I mark as
> expirable messages that are alredy older than expire-age. Those which
> get marked early after receiving and wait as expirable, apparently make
> it nicely to the archive.
> 
> BTW. Judging by diff between emacs-28.2 and master there were very
> little changes in nnmaildir, so I guess the bug (if it's a bug, but hey
> my misconfiguration should be handled better anyway) still exists.
> 
> --8<---------------cut here---------------start------------->8---
> #!/bin/sh
> 
> DIR=$1
> if [ ! -d "$DIR" ]; then
>         echo $DIR does not exist
>         exit 1;
> fi
> 
> FILE=$(date +%s.M%6N)P$$.$(hostname)
> 
> cat >$DIR/tmp/$FILE <<EOF
> From: <42@example.com>
> To: <2137@example.com>
> Subject: test test test
> Date: $(date -d '1 year ago' -R)
> Message-ID: <$(date | sha256sum | cut -f1 -d\ )@example.com>
> 
> EOF
> fortune >>$DIR/tmp/$FILE
> mv $DIR/tmp/$FILE $DIR/new/$FILE
> --8<---------------cut here---------------end--------------->8---

Adding Eric and Lars to the discussion.