From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id d10iC0OvUmCtQAAA0tVLHw (envelope-from ) for ; Thu, 18 Mar 2021 01:39:15 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id 0DFSBkOvUmC+BQAAB5/wlQ (envelope-from ) for ; Thu, 18 Mar 2021 01:39:15 +0000 Received: from mail.notmuchmail.org (nmbug.tethera.net [144.217.243.247]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id D31EF1CA7B for ; Thu, 18 Mar 2021 02:39:14 +0100 (CET) Received: from nmbug.tethera.net (localhost [127.0.0.1]) by mail.notmuchmail.org (Postfix) with ESMTP id 926E626C7C; Wed, 17 Mar 2021 21:39:08 -0400 (EDT) Received: from fethera.tethera.net (fethera.tethera.net [IPv6:2607:5300:60:c5::1]) by mail.notmuchmail.org (Postfix) with ESMTP id 82F9926A27 for ; Wed, 17 Mar 2021 21:39:06 -0400 (EDT) Received: by fethera.tethera.net (Postfix, from userid 1001) id 2726A5FA96; Wed, 17 Mar 2021 21:39:06 -0400 (EDT) Received: (nullmailer pid 1128883 invoked by uid 1000); Thu, 18 Mar 2021 01:39:04 -0000 From: David Bremner To: Gregor Zattler , notmuch , xapian-discuss Subject: Re: bug: chokes on long directory names (was: Re: out of memory on idle machine) In-Reply-To: <20210317194728.GB5561@no.workgroup> References: <20201213151336.GF21521@no.workgroup> <20201213212252.GH21521@no.workgroup> <20201214192251.GA7858@no.workgroup> <20210130085432.GA14025@no.workgroup> <87bld6shrk.fsf@tethera.net> <20210131081638.GA5640@no.workgroup> <87sg6dweds.fsf@tethera.net> <20210207214654.GA2906@no.workgroup> <87o8gqx4d4.fsf@tethera.net> <87k0rex2j9.fsf@tethera.net> <20210317194728.GB5561@no.workgroup> Date: Wed, 17 Mar 2021 22:39:04 -0300 Message-ID: <87blbh5jgn.fsf@tethera.net> MIME-Version: 1.0 Message-ID-Hash: LCWHFBSJXZ6JH74E2F43RHQAP5RKLFG4 X-Message-ID-Hash: LCWHFBSJXZ6JH74E2F43RHQAP5RKLFG4 X-MailFrom: david@tethera.net X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-notmuch.notmuchmail.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; suspicious-header X-Mailman-Version: 3.2.1 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Help: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1616031554; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=sgQSbiGAeWaDUoU9xC39Zix5g4JNVyL9wAuvRQEl6QA=; b=B6Dz903aKd++2qvTQIyR2i2v2RzJGw6+CFOvDdOZjWxyzFhi3DkYKrECqE7RfxYFY2iVvm +ZnblGmJRKy2OCHxh5wCzNkvwewBWG+AO0+rrgPdvdyXc6XF2sNfSqD25hENqzzR5byQpc az6gvryjbhP26TQbOFI1Kybln7Drrg+hlRnSBih8ipV1jJtvqsgeYZUBe1Q6qtVTlWraTw qefKQsFVtAL0iTxXuQkB4gWfUbuLnzWMnEn8PpuvNOYe1hUrkzYGTEHI1MEfYY+D4VhO5g RU+l13d6UY+27mThEm75rxzWNgvAHTpw4arSLG22Kkg+zmExFogGp/yir+h7oA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1616031554; a=rsa-sha256; cv=none; b=FYFJTvYS7A/kdwRfAANIWQuQUbVVnyrIOTL+l7e3t84SirAd1iIm7dCSXXJ4fQMjPUjGlY XWgUXqOyOH5+fFQp+zrmrI8Vn1vWujreVJFhHfl81pZmrWMdi7lKOd+QfzRymPjggkezCf 4hi4sDHbBT0vHoVzlMDQJ5lJGnneywbZO4QO0TZ+0M3eimtDN8GcZUGNqBdvhB4vZddBrj cSAsDT7MpieZsQczhRKz0FrqRc8mABCQ8uqAwiRQFpeHAyIuSQDkGfJlCRZVJAcrgS6E5s NO7XmcnvfNzD6/J013uNMBVF707kLJi9Emu6SqtNmC0YC1Tp+SzVfe+RGMXWSw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 144.217.243.247 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Spam-Score: -2.10 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 144.217.243.247 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Queue-Id: D31EF1CA7B X-Spam-Score: -2.10 X-Migadu-Scanner: scn0.migadu.com X-TUID: ocVToweSAZX+ Gregor Zattler writes: > Hi David, Olly, notmuch and xapian developers, > * David Bremner [11. Feb. 2021]: >> David Bremner writes: >> As a kind of desperation move, you could try bisecting your mailstore, >> to see how small of a set of messages you can duplicate the problem >> with. > > this I did, somehow. I found the culprit: It's a maildir > with one single mail in it. The name of the maildir is > exceptionally long [because generated from a List-Id: > -Header] and the mail arrived at the very day, my notmuch > database corrupted. This maildir alone provokes that every > next notmuch new will rescan all (?) files. Hi Gregor; I am very impressed with your persistence. I suspect it is a bug in notmuch. I don't know all the details yet, but in the normal case the directory name is added to the database prefixed with XDIRECTORY. I noticed this isn't happening in the case of directories 234 or longer. That is roughly the Xapian term limit of 245 characters in total. I'm not sure why the discrepency of one character, but the main point is that notmuch is probably improperly ignoring an error from Xapian when adding these overlong terms. Thanks again for the debugging, I suspect would have never found this bug on my own. David