[PATCH] net_reader: restart on first UID when UIDVALIDITY changes

[PATCH] net_reader: restart on first UID when UIDVALIDITY changes

[Eric Wong]

In other words, treat the same IMAP folder with a different
UIDVALIDITY a completely different folder.  If the UIDVALIDITY
changes, we can start from UID=1 without falling behind or
losing data.  If the UIDVALIDITY gets reset to a previously
known-good message, we can still resume where we left off
before the first UIDVALIDITY change.

This affects public-inbox-watch and "lei import"

One potential downside of this is for rare altid users, but
that's mainly intended for NNTP article numbers which are/were
often publicized; not IMAP UIDs which are rarely publicized.

The other potential downside is bandwidth waste in in the rare
case UIDVALIDITY changes while IMAP folder contents remain
unchanged.  There's no extra storage used due to existing
(v1|v2|lei/store) deduplication mechanisms.

Before this change, we were matching offlineimap behavior and
stopped synching an IMAP folder when its UIDVALIDITY changed.
offlineimap behavior made sense for IMAP <=> Maildir
synchronization since Maildirs had no sense of UIDVALIDITY and
could only rely on name mapping.
---
 lib/PublicInbox/IMAPTracker.pm | 16 ++++++++++++----
 lib/PublicInbox/NetReader.pm   | 11 +++--------
 2 files changed, 15 insertions(+), 12 deletions(-)

diff --git a/lib/PublicInbox/IMAPTracker.pm b/lib/PublicInbox/IMAPTracker.pm
index bcf7af2e..fe813582 100644
--- a/lib/PublicInbox/IMAPTracker.pm
+++ b/lib/PublicInbox/IMAPTracker.pm
@@ -39,12 +39,20 @@ sub dbh_new ($) {
 	$dbh;
 }
 
-sub get_last ($) {
-	my ($self) = @_;
-	my $sth = $self->{dbh}->prepare_cached(<<'', undef, 1);
+sub get_last ($;$) {
+	my ($self, $validity) = @_;
+	my $sth;
+	if (defined $validity) {
+		$sth = $self->{dbh}->prepare_cached(<<'', undef, 1);
+SELECT uid_validity, uid FROM imap_last WHERE url = ? AND uid_validity = ?
+
+		$sth->execute($self->{url}, $validity);
+	} else {
+		$sth = $self->{dbh}->prepare_cached(<<'', undef, 1);
 SELECT uid_validity, uid FROM imap_last WHERE url = ?
 
-	$sth->execute($self->{url});
+		$sth->execute($self->{url});
+	}
 	$sth->fetchrow_array;
 }
 
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index c7b43f01..39129b34 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -397,17 +397,12 @@ sub _imap_fetch_all ($$$) {
 	$r_uidnext //= $mic->uidnext($mbx) //
 		return "E: $uri cannot get UIDNEXT";
 	my $itrk = _itrk($self, $uri);
-	my ($l_uidval, $l_uid) = $itrk ? $itrk->get_last : ();
-	$l_uidval //= $r_uidval; # first time
+	my $l_uid;
+	$l_uid = $itrk->get_last($r_uidval) if $itrk;
 	$l_uid //= 0;
-	if ($l_uidval != $r_uidval) {
-		return "E: $uri UIDVALIDITY mismatch\n".
-			"E: local=$l_uidval != remote=$r_uidval";
-	}
 	my $r_uid = $r_uidnext - 1;
 	if ($l_uid > $r_uid) {
-		return "E: $uri local UID exceeds remote ($l_uid > $r_uid)\n".
-			"E: $uri strangely, UIDVALIDLITY matches ($l_uidval)\n";
+		return "E: $uri local UID exceeds remote ($l_uid > $r_uid)\n";
 	}
 	return if $l_uid >= $r_uid; # nothing to do
 	$l_uid ||= 1;

Re: [PATCH] net_reader: restart on first UID when UIDVALIDITY changes

[Eric Wong]

Eric Wong <e@80x24.org> wrote:
> In other words, treat the same IMAP folder with a different
> UIDVALIDITY a completely different folder.  If the UIDVALIDITY

"as a completely different folder"

Pushed as commit 7632d8f7590daf70c65d4270e750c36552fa9389