From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS6315 166.70.0.0/16 X-Spam-Status: No, score=-3.7 required=3.0 tests=AWL,BAYES_00, RCVD_IN_DNSWL_LOW,SPF_HELO_NONE,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from out03.mta.xmission.com (out03.mta.xmission.com [166.70.13.233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 9DA7D1F461; Tue, 25 Jun 2019 12:02:04 +0000 (UTC) Received: from in02.mta.xmission.com ([166.70.13.52]) by out03.mta.xmission.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.87) (envelope-from ) id 1hfk9D-0008OF-D0; Tue, 25 Jun 2019 06:02:03 -0600 Received: from ip72-206-97-68.om.om.cox.net ([72.206.97.68] helo=x220.xmission.com) by in02.mta.xmission.com with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.87) (envelope-from ) id 1hfk9C-0005oX-M3; Tue, 25 Jun 2019 06:02:03 -0600 From: ebiederm@xmission.com (Eric W. Biederman) To: Eric Wong Cc: meta@public-inbox.org References: <878strvusz.fsf@xmission.com> <20190624163442.xhk6drl7ptnq7i5o@dcvr> <20190624173319.3bb4t3zrieb4k5w2@dcvr> <877e9avbh2.fsf@xmission.com> <20190624234212.j7ghrejyv7ts7y2u@dcvr> Date: Tue, 25 Jun 2019 07:01:54 -0500 In-Reply-To: <20190624234212.j7ghrejyv7ts7y2u@dcvr> (Eric Wong's message of "Mon, 24 Jun 2019 23:42:12 +0000") Message-ID: <878stpub4d.fsf@xmission.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-XM-SPF: eid=1hfk9C-0005oX-M3;;;mid=<878stpub4d.fsf@xmission.com>;;;hst=in02.mta.xmission.com;;;ip=72.206.97.68;;;frm=ebiederm@xmission.com;;;spf=neutral X-XM-AID: U2FsdGVkX19MWiPpwuPVSrIsjUCIVdji1QUSJ6XbMf8= X-SA-Exim-Connect-IP: 72.206.97.68 X-SA-Exim-Mail-From: ebiederm@xmission.com Subject: Re: Q: Did you do something to message number recently? X-SA-Exim-Version: 4.2.1 (built Thu, 05 May 2016 13:38:54 -0600) X-SA-Exim-Scanned: Yes (on in02.mta.xmission.com) List-Id: Eric Wong writes: > "Eric W. Biederman" wrote: >> The add method computes the number using num_for, which uses >> Msgmpa::mid_insert. >> >> Short of the sequence number for msgmap getting scrambled I don't >> see how that can go wrong. Sigh. > > That seems to be what went wrong with "INSERT OR IGNORE" > > cf. https://public-inbox.org/meta/20190624233809.1721-1-e@80x24.org/ Is it INSERT OR IGNORE or is it the autoincrement? As I read the sqlite docs autoincrement is allowed to skip message numbers. Especially in cases where the insert fails. This fix should come with a test case shouldn't it? Such a test case should be pretty straight forward to write. Insert a message multiple times and then insert a new one. I don't recall the transaction constraints on sqlite but I am wondering if instead of INSERT OR IGNORE we should read and the increment num_highwater in the public-inbox code and then we would not need to depend upon autoincrement only incrementing once. Even when the sqlite docs don't promise it will. >> I will have to instrument my client, and see if that will give >> me a clue what is going on. > > I guess your client ends up seeing the same message > over-and-over again like -watch does? It usually doesn't but for dealing with a different imap server I tweaked the code and one of the servers feeding me the last message in the mail box every time I checked the mailbox for new messages. I have also narrowed it down to duplication insertions. I was just about to start looking at what the fix would be. >> I think most of my previous work was all in the index rebuild path. >> Which may be why we don't have a regression test catching this. > > Yeah, this isn't a new bug at all. I didn't notice since I > mainly use w3m or (neo)mutt for NNTP and didn't look too closely > at NNTP article numbers. lynx makes it apparent messages are > missing: lynx news.public-inbox.org/inbox.comp.lib.glibc.alpha It is a little obscure in gnus but I start getting suspcious when I tell it to give me the last 10 messages and I get only one. Eric