From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id WCMcCQY3KGCaZQAA0tVLHw (envelope-from ) for ; Sat, 13 Feb 2021 20:31:02 +0000 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id SFnmBAY3KGAafQAAB5/wlQ (envelope-from ) for ; Sat, 13 Feb 2021 20:31:02 +0000 Received: from mail.notmuchmail.org (nmbug.tethera.net [IPv6:2607:5300:201:3100::1657]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 9E405146DF for ; Sat, 13 Feb 2021 21:31:01 +0100 (CET) Received: from nmbug.tethera.net (localhost [127.0.0.1]) by mail.notmuchmail.org (Postfix) with ESMTP id 28FC02062B; Sat, 13 Feb 2021 15:30:52 -0500 (EST) Received: from mout.gmx.net (mout.gmx.net [212.227.17.20]) by mail.notmuchmail.org (Postfix) with ESMTPS id 26A951FBB5 for ; Sat, 13 Feb 2021 15:30:49 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1613248245; bh=oRpY6xO1CQAyXr6L2SrXAOdCN3O7P56BLL1x7+Z8PxQ=; h=X-UI-Sender-Class:Date:From:To:Subject:References:In-Reply-To; b=c5D5E3RmQJBavG4COGI3A8PRKHok8GiSAffcFecdyt/iulRgcAurMFBd4Tx9hQMDA bSSUh6lWPGpE4+empv+dQtkpOcxBV+fuNX6ftdQ5bCQ5JbLuN50q1Dt3HX2R5MwRgt MIi5J1LohQghUIY3erGMvuIIs+OSMld/0CvrsJoE= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from no.workgroup ([46.94.149.59]) by mail.gmx.net (mrgmx104 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MQ5rU-1lX4eT1mhk-00M0x9; Sat, 13 Feb 2021 21:30:45 +0100 Date: Sat, 13 Feb 2021 21:30:36 +0100 From: Gregor Zattler To: Olly Betts , notmuch Subject: Re: out of memory on idle machine Message-ID: <20210213203036.GB19041@no.workgroup> Mail-Followup-To: Olly Betts , notmuch References: <87zh2hhk15.fsf@tethera.net> <20201213141543.GE21521@no.workgroup> <20201213151336.GF21521@no.workgroup> <20201213212252.GH21521@no.workgroup> <20201214192251.GA7858@no.workgroup> <20210130085432.GA14025@no.workgroup> <87bld6shrk.fsf@tethera.net> <20210131081638.GA5640@no.workgroup> <87sg6dweds.fsf@tethera.net> <20210209043419.GC19807@survex.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20210209043419.GC19807@survex.com> X-Provags-ID: V03:K1:Wxd5CuFDVXAtcruUG/ZqqNrReNU8DusqMt2XsIEVsThJRxDjI32 6Uolud/BQSmuFBgSxP8c/5xA0Ua74hTjMSLqgY0dDWujiyzmOfHdt3wv+wSL4aSp2no89Mn Vnp4AK1LFHTDH8usQT/dA0C8AA6/VmQ+NDtzaCxiTuutbgrUoc9Yb5ljbFk1MhwDV9iFaqJ +fY/umkKWGuYoW6h+MUZA== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:gmwVy7/VKGc=:imWzJUkz1uwDLkUGGTGLc/ /E2lvCDUAf3vgz5vwqVaTjvflFA6365Gw06X2RaY9+tQJ1PekJ/aHHZtQKpRKBuF8oeKc8r9q AJusaiIXgkKgC4/e+nDy3U/mRaZYbWwLcC2xeRzxR0/EYWic4YyzL52ypsuZ1Xwf80XfJluUn bfyM2I5e4Kk/QhjzHgTiw/vgmUsYz3LEA7njxfIpmkUoAFbfq/1uIvtTprKSB5Bw7x+aaMTaN utnA1GGLfUpfwnvWGxrh94aspS4xtzHjhz9ah7Bwb5JBbETYxp9Vy/9mU208t1QrNZSdesLF/ v5GiUgQwy7Fk5Y/t3W9qphuBomxf0t5hYPfLiswkhGXFqoVE5GFU4PTlHsjFFcVA8Q0x8i8LM 4AD4CIi0S2f6HABskDzSKYj92IiF5U1mbwz1WMOiGkVjEfE8ckwmvyzy85Ny1FKqM+X5jPfaC M+BW0od3se6fY3QVxWa+9dxssGLySURRwaZK1ZaJ6nrl2T5BfcF+Q7VNFJm9/g2R5lNa/jNXF SBXNFHAFAW/wrtiSfa+0awYva2Bv1qD7xUzZc5SgTTzUayGoYu+fx8EYQJAH7HaqnZ4lBekFr 1M9ZmORc+HPOtU3XqOslKuWNmsgxWq9EB38ca2aA5AnnTupc4EN661VtP2b5PQshYj4qKllzK F6OZpUMMAue8bZM4iCJ4GARLBWYpoQgtHRK5VxhKhOJqZM3JCaRpAwdij+ZFFp984St55bJW8 sIDqSEiaMObqpPy0yukFkykHUuoQtGMWpb07bivrL/ssfGEkfegJqdbzWQkPTZ3OlRa/q6Qa2 26L8Mmq2hUeGSy/s3hGGjBPBTcB58xU6BTGV1sLzCWGeRgWsloLcqWIldds8hcAEPzxbblGaE wNMvZq3OUFmTqELQKjaA== Message-ID-Hash: POJ6WH3UYSIQTVXEPCYB2ICPWWB32UMP X-Message-ID-Hash: POJ6WH3UYSIQTVXEPCYB2ICPWWB32UMP X-MailFrom: telegraph@gmx.net X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-notmuch.notmuchmail.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; suspicious-header X-Mailman-Version: 3.2.1 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Help: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: 0.63 Authentication-Results: aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=gmx.net header.s=badeba3b8450 header.b=c5D5E3Rm; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmx.net (policy=none); spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 2607:5300:201:3100::1657 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Queue-Id: 9E405146DF X-Spam-Score: 0.63 X-Migadu-Scanner: scn1.migadu.com X-TUID: mnLTtPdG/+8s Hi Olly, notmuch and xapian developers, sorry for late answer, I had problems with the test system: * Olly Betts [09. Feb. 2021]: > On Wed, Feb 03, 2021 at 07:59:43AM -0400, David Bremner wrote: >> Gregor Zattler writes: >>> A Xapian exception occurred finding message: Db block overwritten - are there multiple writers?. >> >> I have included the Xapian list in copy in case that message rings a >> bell. > > There was a bug fixed in 1.4.7 which incorrectly resulted in this error > message, but it seems from the quoted text you're using 1.4.11. yes. >> I guess you know there are not multiple writers in your setup. > > There's a lock file locked by fcntl() which protects against multiple > writers, so someone/something would have need to have deleted that > behind Xapian's back, or else a bug somewhere in the locking code stack. There is no other writer. The system is used only for this test atm, the emails are on a dedicated data partition. > (Aside from that bug, probably the most common case here over time has > been that someone deleted the lock file thinking it's "stale", but it's > not the mere presence of the file that means the lock is held. It's > not at all frequent, but perhaps we should adjust this message to better > reflect that.) > > Have you tried xapian-check on this database? not this time. Yust wait. I have different databases from different runs of notmuch new, "xapian-3" being identical with "xapian-3" grfz@mic:~/Mail/.notmuch$ for name in xapian xapian-3 xapian-2 xapian-1 ; do echo "===== $name"; xapian-check $name ; done > /tmp/xapian-checks 2>&1 results in: ===== xapian docdata: blocksize=8K items=577 firstunused=10 revision=4 levels=1 root=6 B-tree checked okay docdata table structure checked OK termlist: blocksize=8K items=68006 firstunused=20302 revision=4 levels=2 root=687 xapian-check: DatabaseError: Block 17959: Used block also in freelist ===== xapian-3 docdata: blocksize=8K items=577 firstunused=10 revision=4 levels=1 root=6 B-tree checked okay docdata table structure checked OK termlist: blocksize=8K items=68006 firstunused=20302 revision=4 levels=2 root=687 xapian-check: DatabaseError: Block 17959: Used block also in freelist ===== xapian-2 docdata: blocksize=8K items=577 firstunused=10 revision=3 levels=1 root=6 B-tree checked okay docdata table structure checked OK termlist: blocksize=8K items=68006 firstunused=20302 revision=3 levels=2 root=8651 B-tree checked okay termlist table structure checked OK postlist: blocksize=8K items=1971726 firstunused=25177 revision=3 levels=2 root=6651 B-tree checked okay postlist table structure checked OK position: blocksize=8K items=9589984 firstunused=52204 revision=3 levels=2 root=18102 B-tree checked okay position table structure checked OK spelling: Lazily created, and not yet used. synonym: Lazily created, and not yet used. No errors found ===== xapian-1 docdata: blocksize=8K items=371 firstunused=4 revision=2 levels=1 root=2 B-tree checked okay docdata table structure checked OK termlist: blocksize=8K items=32426 firstunused=8650 revision=2 levels=2 root=687 B-tree checked okay termlist table structure checked OK postlist: blocksize=8K items=562665 firstunused=6650 revision=2 levels=2 root=380 B-tree checked okay postlist table structure checked OK position: blocksize=8K items=4359147 firstunused=18099 revision=2 levels=2 root=377 B-tree checked okay position table structure checked OK spelling: Lazily created, and not yet used. synonym: Lazily created, and not yet used. No errors found There are no problems reported on earlier databases although they resulted in reindexing of almost all of the emails and in case of the corrupted xapian database not all file are checked. This is not fixable: grfz@mic:~/Mail/.notmuch$ cp -a xapian xapian-checked grfz@mic:~/Mail/.notmuch$ xapian-check xapian-checked F docdata: B-tree checked okay docdata table structure checked OK termlist: xapian-check: DatabaseError: Block 17959: Used block also in freelist grfz@mic:~/Mail/.notmuch$ xapian-check xapian-checked docdata: blocksize=8K items=577 firstunused=10 revision=4 levels=1 root=6 B-tree checked okay docdata table structure checked OK termlist: blocksize=8K items=68006 firstunused=20302 revision=4 levels=2 root=687 xapian-check: DatabaseError: Block 17959: Used block also in freelist >> Olly Betts mentioned in a different thread that he will build a version >> of xapian 1.4.18 for buster backports, so trying with that is probably a >> good step when it is available. > > Yes - 1.4.18 packages are now in Debian testing, so hopefully I can get > this done soon. OK, I'll wait for that. >> % xapian-delve -1 -A XDIRECTORY ~/Mail/.notmuch/xapian | sort -u > delve.txt > > FWIW, the output should be sorted and unique already (sorted by byte > order, so equivalent to `LC_ALL=C sort`). ok, thanks. Ciao, Gregor -- -... --- .-. . -.. ..--.. ...-.-