From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id kEkJIhdeUmAacwAA0tVLHw (envelope-from ) for ; Wed, 17 Mar 2021 19:52:55 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id aAG/HRdeUmAyfwAAB5/wlQ (envelope-from ) for ; Wed, 17 Mar 2021 19:52:55 +0000 Received: from mail.notmuchmail.org (nmbug.tethera.net [144.217.243.247]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 066561838E for ; Wed, 17 Mar 2021 20:52:55 +0100 (CET) Received: from nmbug.tethera.net (localhost [127.0.0.1]) by mail.notmuchmail.org (Postfix) with ESMTP id 49F8820615; Wed, 17 Mar 2021 15:52:46 -0400 (EDT) X-Greylist: delayed 305 seconds by postgrey-1.36 at nmbug; Wed, 17 Mar 2021 15:52:43 EDT Received: from mout.gmx.net (mout.gmx.net [212.227.15.18]) by mail.notmuchmail.org (Postfix) with ESMTPS id 40A2420019 for ; Wed, 17 Mar 2021 15:52:43 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1616010761; bh=/oiF29tLAOw5TnBP93dxMi7CXT16eChkQKuB3M6LSQ0=; h=X-UI-Sender-Class:Date:From:To:Subject:References:In-Reply-To; b=HZ5+Q1pKnr24HTwKldgxnJAJxd76mz7Rm0ihNjbDg2duyuay/jqffoETP5mXb0c7T bCTYV6k/7Ipufb0FsK7j7T+eKZ7mDCrclLvg/RrDZtv7WI5dfjGNQ3EpBBwqA94wN+ Pu4ypU+sMv05R467L+iNaw+W7fjRaTYVtPvrPexY= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from no.workgroup ([84.185.97.89]) by mail.gmx.net (mrgmx004 [212.227.17.190]) with ESMTPSA (Nemesis) id 1Mr9Bk-1m0ui62xr4-00oENL; Wed, 17 Mar 2021 20:47:36 +0100 Date: Wed, 17 Mar 2021 20:47:28 +0100 From: Gregor Zattler To: notmuch , xapian-discuss Subject: bug: chokes on long directory names (was: Re: out of memory on idle machine) Message-ID: <20210317194728.GB5561@no.workgroup> Mail-Followup-To: notmuch , xapian-discuss References: <20201213151336.GF21521@no.workgroup> <20201213212252.GH21521@no.workgroup> <20201214192251.GA7858@no.workgroup> <20210130085432.GA14025@no.workgroup> <87bld6shrk.fsf@tethera.net> <20210131081638.GA5640@no.workgroup> <87sg6dweds.fsf@tethera.net> <20210207214654.GA2906@no.workgroup> <87o8gqx4d4.fsf@tethera.net> <87k0rex2j9.fsf@tethera.net> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <87k0rex2j9.fsf@tethera.net> X-Provags-ID: V03:K1:NSOezbvhL7id6Jzn6qEcabLg0gCAZcHoKWmNyHeayx0zxmk9Yt9 TNjMX8WlPfTdIaMfsKGVEwpk7/eZHkhW+ARPYQZl2rNx0AusbdzDU7wdzp/esnxuiyJrayw P35G3uSr5P9PeJOXUPaTMGy/JJ5Os30obFw7vM5ROb5D79Ru+S3daLzArlgH70ANhu+r8LM 9pC/NRSLJLEIz4tDF5ANQ== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:yVedsq2opyw=:znAwcfYzm/isSXftpfwbh7 zfIyvOOtaxGpaNhMUbnYOPfNT152LNgQa7r+DN08LAX08aVjI0Tt0LXFI8kP4kJupSjSOTmcf khXnZFzoC363+jTwp1EC7mMzWDNiH7FGXZhpGutODfJmrAh1l0vgbLKKSUJYHz03keoE7lGeg v1Ufuo40vdEJGn2VQPqvIKmrGRufcg50Vr3lnLqeWjnb/YEZmMXUeu6WmekDtKnkmaMXPZe42 v3BgIUzJ2qcmJ10LdiCmOevbBIyq2YqQ/hAnyzmq8oARF7Tc9exCfN7fDDoYZqXdE4E1L2TKk WJk1RizJfnpJYBvjU2pDRG/NG3vYWSraO0NIWV+nmInVAbW5mQzgUKDec0JpJU2/CeQExSB+J dksOEzkCjh8D7qUBTK26aPw2g52V8crWTiliblZfPxDZgfM4LWI4UKE7epxn0mJRImvq7AmD0 sg8pc1RODqAyYUs/7CkNHEM4WeD1vpzgT2zTfp1oEzdAFWpEDd/DWtCOkOCEboIc9/TsF3fD2 3SbPRw5mBuRRmxy6TIHfT/PVbcv1o1CDaorX3j8AJWKmPPPYPberxGVWpm+GwXnAk9N70uB1c ry5ox197FXSDv7wFLGOfgfrdgv7E06gf+hkFJHplAg0CMjbaJssZnHtcEJ/5OF1KtHgTXIcUe eoNf44RfXMj8gW+mXZH5Y3T3d2xC+oHHOm1ovTDAWbT9JCHygHUW7QxgQT8351C3H2V4H7hz0 RWM0ceOWA0lqsrudZ5Wyktz+aCcrZEBGfyp5C7kkxaDKQfjGmcZuLRUWyKfRNU2CBwMrmRG+1 02oA1ZTZ5Ora3YoBp8jrFtzxzw7pUf4pJjBbtgZhvunZlTfDAh6RruGoIQ/BE/B6CgjDPu4xk DBtJ7FCETqYxbtpyxzIe+N4fsOWecgJ1Y6qY88piGDssJ8lT1UhsUhK9ODkfQKQ9bOuTIyhre QBtCRaWoYXpcy4TN0qzkssWA2m0eu6LIq+kL/eSGy1F6kSa4tZpcLpTbNEcQDWRtx64iRhyui z81BJKJBNMo0WWtklA1MucFhA565JmcVxW504mn36C6u5971zT670wMgFXTStQSbEU28DnDMu yS848UrwUD9RaP7W02qwk6m4srkHY4oYoOzvD8o8R+t/mb6FsEU0vaIRNiK1Mu+/VK6gfG7xb lR6tgdPNkPcDr7TdOePrt6+a9TxlB0jrchSRH8ykcwizd+YSCqI8Pzb3oj44Ozew1x2+p3xF+ n8hTz6EPkl1h70N3sVPvruiJGtYiFEfXToZVJ/g== Message-ID-Hash: KT5OIRQO5KUR7DNXICD5TKOMPQAYIEAT X-Message-ID-Hash: KT5OIRQO5KUR7DNXICD5TKOMPQAYIEAT X-MailFrom: telegraph@gmx.net X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-notmuch.notmuchmail.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; suspicious-header X-Mailman-Version: 3.2.1 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Help: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1616010775; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=BIoyxbnx7aiQ9dTfjFh1vmT9i4DLiXLaYnpJPuJHVy4=; b=ESpUmE0uSHPrOebL8fTl0jvHz6qwoItG+/8RG3+VO3M4yOqq+WJ/TuLDn0EXPeGREd5uCB zxtY1qTQqmPnoX6rl3pFKMwEc1Vn2TMoaNMOA1N+qIb0Bl3JQNyESyvVpPfy2+gGyaMXMW w9Q1b9HMtjxP/+cHKik/kBIR4Ax/Jc8I+Pw8soyz+dkjkojhFrMvF73q6Neaj0dkTC3LXn GBhHy67rvQpwWAkkf6y3XsJ2VnRZQH1QcH6lXkRo0yhPTVNRA+mWOBpvKxOf0B8rQCJa/4 1pyPHuJoNYw617MYd4e2PxwTXIXGsqbYBdBHQ4b6DrklXskUNZhada6XXA4ivA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1616010775; a=rsa-sha256; cv=none; b=rzTLIQs4JoxotwfR69cO8QNQu21iK/upByOIv3nX/ja3/MUJL1kp4d/4X8paUn9IdL3NPz HPGjQOVjWv/PtHcmEzjMa8TD0forMz9iaXwqSdFaIPL7AG9Cs3agWohwzffPbXPtYzIRdK usvtgyNG8eSXa7yUa2qJVDU64ayGp45ZkjXxoUOXGkiXkDLYUCqztpjU3TJzjkBfVslHRf N2dtlQeV5/V7iVeUwC2pdYT1klHVMSBrDUj54+OABixXZ3QVEYRhsf0Ss8x4Lu1/M3kXVv lT2oxW+SyzKXOHoex3aRrULqw8OsfyDNsABia+cOevpcaeEiuG9cx7/srFsgrw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=gmx.net header.s=badeba3b8450 header.b=HZ5+Q1pK; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmx.net (policy=none); spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 144.217.243.247 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Spam-Score: -0.89 Authentication-Results: aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=gmx.net header.s=badeba3b8450 header.b=HZ5+Q1pK; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmx.net (policy=none); spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 144.217.243.247 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Migadu-Queue-Id: 066561838E X-Spam-Score: -0.89 X-Migadu-Scanner: scn0.migadu.com X-TUID: s3w9ssJMy5ay Hi David, Olly, notmuch and xapian developers, * David Bremner [11. Feb. 2021]: > David Bremner writes: > As a kind of desperation move, you could try bisecting your mailstore, > to see how small of a set of messages you can duplicate the problem > with. this I did, somehow. I found the culprit: It's a maildir with one single mail in it. The name of the maildir is exceptionally long [because generated from a List-Id: -Header] and the mail arrived at the very day, my notmuch database corrupted. This maildir alone provokes that every next notmuch new will rescan all (?) files. Then I tried to only index this maildir, it showed the same strange re-indexing but even when running notmuch new for a while in a loop (>1000 times), the database showed no corruption. When instead I shorten the name of the maildir to three characters with the very same email file in it, nothing happens, it indexes the file once and not again. Then I prolonged the name of the file instead of the directory and even with the longest possible filename (or path?) /home/grfz/Mail/nuk/new/1607641473.31514_2.no1607641473.31514_2.no1607641473.31514_2.no1607641473.31514_2.no1607641473.31514_2.no1607641473.31514_2.no1607641473.31514_2.no1607641473.31514_2.no1607641473.31514_2.no1607641473.31514_2.no1607641473.31514_2.no16076414734160.14_2.no notmuch has no problem indexing this and not to reindex it in the next run. So notmuch or xapian (I don't know) chokes on extreme long directory names. I consider this to be a bug. My scripts create this long names from List-Id and some such. The one which triggered the problems is from an online shop: u+mq6tamjqhe3cm2j5giydembrgiytamrtga2deojogexdsmzygm4egnbuifatcnrsgazdejjugbzgkylmfvxw43djnzsxg2dpoaxgizjgna6ton3bg4zdsobsgmytczlcme3dentehaydmnjxmy4doyrwha4tgobgoi6xizlmmvtxeylqnastimdhnv4c43tfoqthipldovzxi33nmvzhgllxmvwgg33nmu@real-onlineshop.de/ Since, as I tested, this can be reproduced with the simplest of email in a maildir with an extremly long name, I do not attach the maildir in question. But if anyone wants it I can send it. I then had a look at other long directory names and there is another one which also triggers the problem, it also has only one email in it and arrived on 12th of January: u+mq6wcodfgmygcjtjhuzdamrrgaytemjrhe2dqmbqfyys4mbxgazugnbsie3doobsgfcdmobfgqygg5ltorxw2zlsomxgo2lunrqweltdn5wsm2b5mu3tkmddhbrdoyrwgvsgeobymi2dszbtg4zdamztmm4dsmzvgjssm4r5orswyzlhojqxa2bfgqygo3lyfzxgk5bgoq6xa4tjozqwg6i@customers.gitlab.com Since I removed both on my laptop, notmuch new works again, yeah! Now I will have a look on my .procmailrc. Thanks for your attention, thanks for notmuch and for xapian, Grgeor -- -... --- .-. . -.. ..--.. ...-.-