Re: non-deterministic behaviour of new.ignore (regexp) test

[Jani Nikula <jani@nikula.org>]

On Sat, 28 Apr 2018, David Bremner <david@tethera.net> wrote:
> For me the following seems to consistently fail after between 30 and 500
> attempts
>
>     export NOTMUCH_TEST_QUIET=yes; count=0; while ./T050-new.sh; do (( count++ )); echo $count; done

I believe this happens because the directory mtime is unchanged from the
previous scan in the test, and we skip the directory before we could
ignore the files. Quoting add_files():

    /* If the directory's modification time in the filesystem is the
     * same as what we recorded in the database the last time we
     * scanned it, then we can skip the second pass entirely.
     *
     * We test for strict equality here to avoid a bug that can happen
     * if the system clock jumps backward, (preventing new mail from
     * being discovered until the clock catches up and the directory
     * is modified again).
     */

I can't reproduce if I add this to the test:

diff --git a/test/T050-new.sh b/test/T050-new.sh
index 9025fa7aa63e..0db76f47130b 100755
--- a/test/T050-new.sh
+++ b/test/T050-new.sh
@@ -260,6 +260,7 @@ output=$(NOTMUCH_NEW 2>&1)
 test_expect_equal "$output" "No new mail."
 
 test_begin_subtest "Ignore files and directories specified in new.ignore (regexp)"
+touch "${MAIL_DIR}" # force rescan of the top level directory
 notmuch config set new.ignore ".git" "/^bro.*ink\$/" "/ignored.*file/"
 output=$(NOTMUCH_NEW --debug 2>&1 | sort)
 test_expect_equal "$output" \

---

However, I'm not sure even that is enough if all this happens in the
same second. I think the way notmuch new is written, it may skip as long
as it ensures a subsequent scan will catch the modified files:

    /* If the directory's mtime is the same as the wall-clock time
     * when we stat'ed the directory, we skip updating the mtime in
     * the database because a message could be delivered later in this
     * same second.  This may lead to unnecessary re-scans, but it
     * avoids overlooking messages. */

I think we can make the problem less likely with the touch, but as
everything gets faster, we might hit this more and more. One approach
might be a notmuch new --force option that would rescan all directories
regardless of mtimes. We could use this for testing (except when we're
testing the optimization).

BR,
Jani.