From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 3A6CB6DE1025 for ; Fri, 28 Jun 2019 10:26:09 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org X-Spam-Flag: NO X-Spam-Score: -1.167 X-Spam-Level: X-Spam-Status: No, score=-1.167 tagged_above=-999 required=5 tests=[AWL=-0.467, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FAKE_REPLY_C=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001] autolearn=disabled Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id my9nxdBw-xq5 for ; Fri, 28 Jun 2019 10:26:08 -0700 (PDT) X-Greylist: delayed 578 seconds by postgrey-1.36 at arlo; Fri, 28 Jun 2019 10:26:08 PDT Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) by arlo.cworth.org (Postfix) with ESMTPS id 36C756DE0F9E for ; Fri, 28 Jun 2019 10:26:08 -0700 (PDT) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 8A8632026A; Fri, 28 Jun 2019 13:16:29 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Fri, 28 Jun 2019 13:16:29 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; bh=SwiCL0rqikppIVcpypLirsy+1M4tI01TUp/m/zM0Gxo=; b=LaEKRryR qJTz3+BLe26qxAaeflsCP3BJ1PyFQR9+x+ys58Bq6Ce4DR7355j63MtRopRPaGRN bR30nPw/yEyoAqiUjS7C3rjH09XCoJHVrEF6Amz7I1SA7uN0kvX23F9ijmNcKO/b H7Yl/N2dg4e+vz7wDF8PSJApOyr2eWWCo7829rj5EECOPLJvaD2EGvUcr8gYbFSm NTAaUDsyIljc7zlxDrdd8XgMZoRzGi+TD2ToB7XpiGUFyFU361KqaeucGO3jYlIc poajmqSfdLcPA4Q6GlJGE8nQlbL9/aJdsHppPZrhWjjNZi5US6Snc/Re7XrQjB+I TY4trN7zhCD/mw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduvddrvddtgdduudefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkgggtugfgjggfsehtke ertddtredunecuhfhrohhmpeetlhhvrghrohcujfgvrhhrvghrrgcuoegrlhhvhhgvrhhr vgesrghlvhhhrdhnohdqihhprdhorhhgqeenucfkphepudeltddruddvuddrvdelrdefne curfgrrhgrmhepmhgrihhlfhhrohhmpegrlhhvhhgvrhhrvgesrghlvhhhrdhnohdqihhp rdhorhhgnecuvehluhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: from nimloth.alvh.no-ip.org (unknown [190.121.29.3]) by mail.messagingengine.com (Postfix) with ESMTPA id 662EA380075; Fri, 28 Jun 2019 13:16:28 -0400 (EDT) Received: by nimloth.alvh.no-ip.org (Postfix, from userid 1000) id 3E9EA1207B4; Fri, 28 Jun 2019 13:16:26 -0400 (-04) Date: Fri, 28 Jun 2019 13:16:26 -0400 From: Alvaro Herrera To: Alexei Gilchrist Cc: notmuch@notmuchmail.org Subject: Re: notmuch ignoring alot of emails Message-ID: <20190628171626.GA20853@alvherre.pgsql> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <9359F7FC-94C5-48CA-B4C9-2FEB2C7EC398@runbox.com> User-Agent: Mutt/1.9.4 (2018-02-28) X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jun 2019 17:26:09 -0000 On 2019-Mar-23, Alexei Gilchrist wrote: > When I run notmuch I get a bunch (hundreds) of emails that are ignored with: > > Note: Ignoring non-mail file: ... > > The files are valid maildir files but have a paragraph somewhere in the body > where someone has written "From ". Yeah, that happens too when you attach patches generated with git format-patch as plain text; this is extremely common in the pgsql-hackers@lists.postgresql.org mailing list (you can download an mbox from there for any month, convert it to a maildir, and give the resulting maildir to notmuch -- you'll likely find a few dozen emails that fail parsing). This is a very annoying problem for me, see 201901181607.4rba4c5uyimv@alvherre.pgsql in this list earlier this year. I worked around it by patching _notmuch_message_file_parse in lib/message-file.c to set is_mbox = false unconditionally; but that's not a real solution (and hence I didn't post as a patch here), and it explodes real good if you have an actual mbox in the directory where the mail is (since after the hack it won't skip it anymore). I think a real solution is to parse the message header, look for the Content-Length, and determine mbox-ness by looking for "From" only past that many bytes; that seems to match what other mail parsing tools do. However, I haven't gotten around to doing that. -- Álvaro Herrera 39°49'30"S 73°17'W "La experiencia nos dice que el hombre peló millones de veces las patatas, pero era forzoso admitir la posibilidad de que en un caso entre millones, las patatas pelarían al hombre" (Ijon Tichy)