From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 4080F6DE0C3B for ; Mon, 21 Jan 2019 12:03:00 -0800 (PST) X-Virus-Scanned: Debian amavisd-new at cworth.org X-Spam-Flag: NO X-Spam-Score: -1.316 X-Spam-Level: X-Spam-Status: No, score=-1.316 tagged_above=-999 required=5 tests=[AWL=-0.615, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001] autolearn=disabled Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pk8NFmB95A1L for ; Mon, 21 Jan 2019 12:02:58 -0800 (PST) X-Greylist: delayed 531 seconds by postgrey-1.36 at arlo; Mon, 21 Jan 2019 12:02:58 PST Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) by arlo.cworth.org (Postfix) with ESMTPS id 4C1D66DE0C3A for ; Mon, 21 Jan 2019 12:02:58 -0800 (PST) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 837D324BF4; Mon, 21 Jan 2019 14:54:04 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Mon, 21 Jan 2019 14:54:04 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; bh=9/Aql18BzW4fHkOiAFSXnquGsKigBN9lUFi6RN81kXI=; b=aaHeIPwI MjQpK4jdhNdo2lhw316ijRBA4vYmKf1z+bmdS+axdmRJrMwOvBAOa63YWCeYVmt6 pgMNZLG1UgUX4LHJee7yAEPIq+yyTzqVcc6YpuflCxi/L2hnCNSk0hoQx6+5xNt4 iRUxzxW/BpmXlsE16x+RMmXC0j+xkiKQ9P2jdzNd13k2eYyvmA2/W/NU49NZ7o87 pn9NPIfrnG67sXW7oKAl4rcLORJI9KKJOxej3pL3/5rKDp/QyEr5nhXpAAXDd97D ogvm1uH+pKUBbWefOu4IG+uK7mEn/GvAv/yjDIN2BIUCsd5TtuvCa3LO7KJGznzY BUYNSOjbuI72tA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedtledrheeigddufedvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfquhhtnecuuegrihhlohhuthemucef tddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjughrpeffhffvuffkgg gtugfgjggfsehtkeertddtredunecuhfhrohhmpeetlhhvrghrohcujfgvrhhrvghrrgcu oegrlhhvhhgvrhhrvgesrghlvhhhrdhnohdqihhprdhorhhgqeenucffohhmrghinhepvd hnughquhgrughrrghnthdrtghomhdpnhhothhmuhgthhhmrghilhdrohhrghdpghhithhh uhgsrdgtohhmpdhivghtfhdrohhrghenucfkphepudeltddruddvuddrvdelrdefnecurf grrhgrmhepmhgrihhlfhhrohhmpegrlhhvhhgvrhhrvgesrghlvhhhrdhnohdqihhprdho rhhgnecuvehluhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: from alvin.alvh.no-ip.org (unknown [190.121.29.3]) by mail.messagingengine.com (Postfix) with ESMTPA id 8776710085; Mon, 21 Jan 2019 14:54:02 -0500 (EST) Received: by alvin.alvh.no-ip.org (Postfix, from userid 1000) id EFF49A74; Mon, 21 Jan 2019 16:53:58 -0300 (-03) Date: Mon, 21 Jan 2019 16:53:58 -0300 From: Alvaro Herrera To: David Bremner Cc: notmuch@notmuchmail.org Subject: Re: BUG: "notmuch insert" fails with "Delivery of non-mail file" Message-ID: <201901211953.q5nxxkrlghgp@alvherre.pgsql> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <87pnssqzpf.fsf@tethera.net> User-Agent: NeoMutt/20180716-1013-2c1e88 X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jan 2019 20:03:00 -0000 Hi David, thanks for replying. On 2019-Jan-19, David Bremner wrote: > Alvaro Herrera writes: > > > In my read of the code ultimately comes from > > g_mime_parser_construct_message rejecting the message. > > I reported this to GMime, and they said that the problem is that notmuch > > insert is using the mbox mode: > > https://github.com/jstedfast/gmime/issues/58 > > (Sample email is attached there). > > This issue (or a related one) has come up before > > https://nmbug.notmuchmail.org/nmweb/search/postfix+mbox > > Generally it seems to be caused by tools that add mbox 'From ' headers, > without actually mbox escaping the file. We haven't yet reached > consensus on a good solution (generally people just want to fix their > own mail, which is understandable). A workaround discussed in the > messages I reference above is to strip the 'From ' header before passing > to notmuch-insert. Perhaps some scholar of the RFCs can convince us that > that is "always" the right thing for notmuch insert to do. I'm not sure I follow. As I understand, notmuch does not work with mboxes, only with maildirs, so the behavior of splitting emails at "From " is not strictly necessary, since one file always equals one message. As for RFC scholarship, I spent some time looking at https://tools.ietf.org/html/rfc5322 to see if it defined any sort of message separator ... but as far as I can tell, it only defines what does a valid message looks like. It doesn't say where does one message end. On the other hand, in my world, it's been quite a while since 'From ' was considered a useful message separator. This stopped being true in a pretty extensive way when git-format-patches messages started being posted as attachments. But even before that, MUAs stopped adding the ">" at the start of a "From " line in human-written text. Nowadays what really governs the split is the Content-Length header, from the MIME definitions. Most tools do not escape lines starting with 'From ' anymore. As far as I can tell, this is defined by RFC-2049, https://tools.ietf.org/html/rfc2046#section-5.1.1 which states that the implementation must look for the "boundary delimitir line". Stopping at a "From " line before finding the boundary delimiter line would be a mistake, in my reading. > > As far as I can tell, this is all coming from > > _notmuch_message_file_parse() which sets the is_mbox flag when it sees > > the "^From " line at the start of the file ... which kinda makes sense > > in general terms, but for notmuch-insert I think that's the wrong thing > > to do. Maybe a solution is to pass a flag down from notmuch-insert.c's > > add_file all the way down to _notmuch_message_file_parse telling it not > > to treat the file as an mbox. > > I'd be worried about letting notmuch-insert deliver messages that > notmuch-new would not be able to parse. In particular we'd like to keep > the property that a Maildir + the output of notmuch-dump should be > enough to completely recover the notmuch database. Hmm, that's a good point -- I assume that notmuch-new should be patched similarly so that those messages are valid there too. So maybe the solution (given that, as I said above, Notmuch does not appear to handle mboxes at all) is to just set the mbox flag to false completely ... -- Álvaro Herrera PostgreSQL Expert, https://www.2ndQuadrant.com/