Unexpected output of "notmuch new --quiet"

Unexpected output of "notmuch new --quiet"

[Ralph Seichter]

Hi folks,

I can't help but wonder why text like

  Note: Ignoring non-mail file: /home/foo/.maildir/.private/dovecot-uidlist
  Note: Ignoring non-mail file: /home/foo/.maildir/.work/dovecot.index.log
  0022

is written when I run "notmuch new --quiet"?

Only error conditions should be reported to stderr, so why are lines 1
and 2 written to stderr? Notes are not errors. Also, why is line #3,
which represents the current umask value, written to stdout?

I'm aware that the man page states "--quiet: Do not print progress or
results.", but the output shown above is still unexpected. Perhaps
somebody here can enlighten me. ;-)

-Ralph

Re: Unexpected output of "notmuch new --quiet"

[David Bremner]

Ralph Seichter <abbot@monksofcool.net> writes:

> Hi folks,
>
> I can't help but wonder why text like
>
>   Note: Ignoring non-mail file: /home/foo/.maildir/.private/dovecot-uidlist
>   Note: Ignoring non-mail file: /home/foo/.maildir/.work/dovecot.index.log
>   0022
>
> is written when I run "notmuch new --quiet"?
>
> Only error conditions should be reported to stderr, so why are lines 1
> and 2 written to stderr? Notes are not errors.

They could quite reasonably be retitled 'Warning' since they sometimes
indicate a disagreement between notmuch and the MDA as to how mail files
should be formatted on disk).  Your case just happens to be on the
harmless end.  I could imagine more levels of quiet, but I don't know if
it is worth the trouble.

> Also, why is line #3, which represents the current umask value,
> written to stdout?

I can duplicate that here. Could it maybe a wrapper script or hook
invoked by notmuch new?

Re: Unexpected output of "notmuch new --quiet"

[Ralph Seichter]

* David Bremner:

> I could imagine more levels of quiet, but I don't know if it is worth
> the trouble.

I currently redirect stderr to /dev/null because the actual number of
these "ignoring non-mail file" notices is much higher, but that means I
don't get to see actual errors which is quite a disadvantage.

An option like --veryquiet or --errorsonly would be helpful, but I can
understand you asking if it would be worth the effort. For somebody like
me who uses Dovecot as a backend it would be. ;-)

> Could it maybe a wrapper script or hook invoked by notmuch new?

Ah! The post-new hook called umask and caused the output. Thanks.

-Ralph

Re: Unexpected output of "notmuch new --quiet"

[David Bremner]

Ralph Seichter <abbot@monksofcool.net> writes:

> * David Bremner:
>
>> I could imagine more levels of quiet, but I don't know if it is worth
>> the trouble.
>
> I currently redirect stderr to /dev/null because the actual number of
> these "ignoring non-mail file" notices is much higher, but that means I
> don't get to see actual errors which is quite a disadvantage.
>

Unfortunately its not very easy for notmuch to distinguish between
different kinds of parsing failures (at least, not easy to do so
quickly). The recommended solution for things like dovecot index files
is to set to set "new.ignore" (see notmuch-config(1)).
 
> An option like --veryquiet or --errorsonly would be helpful, but I can
> understand you asking if it would be worth the effort. For somebody like
> me who uses Dovecot as a backend it would be. ;-)

I guess if you have a simple way of distinguishing the cases which you
want to consider as errrors, we can revisit the idea.

Re: Unexpected output of "notmuch new --quiet"

[Ralph Seichter]

* David Bremner:

> I guess if you have a simple way of distinguishing the cases which you
> want to consider as errrors, we can revisit the idea.

Personally, I'd go with these decreasing levels of severity:

  Fatal: Execution must stop immediately to prevent damage, error
  message may or may not be displayed before exiting the process.

  Error: Serious trouble, program may or may not be able to recover,
  user intervention required.

  Warning: Minor trouble, program will recover without user
  intervention.

  Notice: Inform the user of a less-than-ideal situation (non-mail file
  found, deprecated functionality used, etc). Program will ignore the
  cause and carry on.

I don't know if this is a fit for Notmuch, but it would be great if I
could prevent warnings and notices from being reported by using a
command line option.

-Ralph

Re: Unexpected output of "notmuch new --quiet"

[David Bremner]

Ralph Seichter <abbot@monksofcool.net> writes:

> * David Bremner:
>
>> I guess if you have a simple way of distinguishing the cases which you
>> want to consider as errrors, we can revisit the idea.
>
> Personally, I'd go with these decreasing levels of severity:
[snip]
> I don't know if this is a fit for Notmuch, but it would be great if I
> could prevent warnings and notices from being reported by using a
> command line option.
>

Just to be clear, I was referring in the question of deciding for
specific messages, which ones are serious and which ones are not. Are
there warnings that you want to suppress that are not handleable with
the new.ignore facility?

d

Re: Unexpected output of "notmuch new --quiet"

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 1068 bytes --]

On Thu 2019-04-18 15:52:19 +0200, Ralph Seichter wrote:
> Personally, I'd go with these decreasing levels of severity:
>
>   Fatal: Execution must stop immediately to prevent damage, error
>   message may or may not be displayed before exiting the process.
>
>   Error: Serious trouble, program may or may not be able to recover,
>   user intervention required.
>
>   Warning: Minor trouble, program will recover without user
>   intervention.
>
>   Notice: Inform the user of a less-than-ideal situation (non-mail file
>   found, deprecated functionality used, etc). Program will ignore the
>   cause and carry on.

Why is "non-mail file found" in my maildir such a low priority?  If
there's a malformed message in my maildir i want to know about it as
soon as possible, that could be an important e-mail that some other part
of my mail ecosystem mangled that i could de-mangle to have access to it
if only i know to look there?

Seems like using new.ignore to ignore the files that you know are
supposed to be there but aren't mail is a cleaner way to go.

   --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Unexpected output of "notmuch new --quiet"

[Ralph Seichter]

* David Bremner:

> Are there warnings that you want to suppress that are not handleable
> with the new.ignore facility?

Sorry, I misunderstood your question, and new.ignore is sufficient for
my current purposes:

  [new]
  ignore=//dovecot[[:punct:]]/;/(maildirfolder|subscriptions)$/;

Thank you for pointing this feature out to me.

-Ralph

Re: Unexpected output of "notmuch new --quiet"

[Ralph Seichter]

* Ralph Seichter:

> ignore=//dovecot[[:punct:]]/;/(maildirfolder|subscriptions)$/;

Odd. I am currently adding this entry to some users' config files, and
the expression //dovecot[[:punct:]]/ does not seem to work in all
cases. Based on notmuch-search-terms(7) I assumed that Notmuch used
POSIX.2 regular expressions, does it not?

-Ralph

Re: Unexpected output of "notmuch new --quiet"

[Ralph Seichter]

* Daniel Kahn Gillmor:

> Why is "non-mail file found" in my maildir such a low priority?

Because I do not agree with Notmuch's notion of what a non-mail file
is. All files the Dovecot server uses for its own metadata are placed
outside of "cur", "new", and "tmp".

My understanding of the Maildir structure is that only these three
subfolders in a given directory may contain mail files, so I expect
Notmuch, as a default behaviour, to ignore whatever is stored outside
of these subfolders.

-Ralph

Re: Unexpected output of "notmuch new --quiet"

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 1405 bytes --]

On Fri 2019-04-19 15:55:27 +0200, Ralph Seichter wrote:
> * Daniel Kahn Gillmor:
>
>> Why is "non-mail file found" in my maildir such a low priority?
>
> Because I do not agree with Notmuch's notion of what a non-mail file
> is. All files the Dovecot server uses for its own metadata are placed
> outside of "cur", "new", and "tmp".
>
> My understanding of the Maildir structure is that only these three
> subfolders in a given directory may contain mail files, so I expect
> Notmuch, as a default behaviour, to ignore whatever is stored outside
> of these subfolders.

Hm, notmuch supports trees of maildirs as well, and nested maildirs,
none of which are part of the maildir spec aiui.

Also, as far as maildir goes, i think notmuch probably *shouldn't* be
looking at stuff in …/tmp/ -- right?  otherwise it's liable to
occasionally access some half-written files.

I've personally done the (maybe bad) thing where i just drop a folder
full of RFC5322 .eml files anywhere inside my message store and run
"notmuch new" to ingest them.  I'd be sad if those files went away, but
perhaps you're right that we should only be looking at are things within
any cur/ and new/ subfolders within the message archive.

That'd be an easy fix if there was no deployed base to worry about.  Any
thoughts about how to make such a transition safe for existing users
like me?

    --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Unexpected output of "notmuch new --quiet"

[David Bremner]

Ralph Seichter <abbot@monksofcool.net> writes:

> * Ralph Seichter:
>
>> ignore=//dovecot[[:punct:]]/;/(maildirfolder|subscriptions)$/;
>
> Odd. I am currently adding this entry to some users' config files, and
> the expression //dovecot[[:punct:]]/ does not seem to work in all
> cases. Based on notmuch-search-terms(7) I assumed that Notmuch used
> POSIX.2 regular expressions, does it not?

I think I can duplicate the problem you see, and it's not about POSIX.2,
but rather that the paths are stored in the database (and match)
relative to the top level Maildir. So the leading '/' isn't there to
match for top level files.

This version seems to work for me.

     ignore=/(^|/)dovecot[-.]/;/(maildirfolder|subscriptions)$/;

Maybe regexp experts can suggest the canonical way to do this;
potentially we could add another example to the docs. 

Re: Unexpected output of "notmuch new --quiet"

[David Bremner]

Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:

> I've personally done the (maybe bad) thing where i just drop a folder
> full of RFC5322 .eml files anywhere inside my message store and run
> "notmuch new" to ingest them.  I'd be sad if those files went away, but
> perhaps you're right that we should only be looking at are things within
> any cur/ and new/ subfolders within the message archive.
>

notmuch(1) is on your side:

       The mail directory you specify can contain any number of
       sub-directories and should primarily contain only files with
       individual email messages (eg. maildir or mh archives are
       perfect). If there are other, non-email files (such as indexes
       maintained by other email programs) then notmuch will do its best
       to detect those and ignore them.

I guess my initial reaction is that the tradeoff discussed here (less
functionality for better/fewer warnings) doesn't seem great.

d

Re: Unexpected output of "notmuch new --quiet"

[Ralph Seichter]

* David Bremner:

> So the leading '/' isn't there to match for top level files.

That's unexpected, so quite in tune with the thread subject. ;-)

> ignore=/(^|/)dovecot[-.]/;/(maildirfolder|subscriptions)$/;

I'll try that, thanks.

-Ralph

Re: Unexpected output of "notmuch new --quiet"

[Ralph Seichter]

* Daniel Kahn Gillmor:

> as far as maildir goes, i think notmuch probably *shouldn't* be
> looking at stuff in …/tmp/ -- right?

You're right, I don't see why Notmuch would peek into "tmp".

> I'd be sad if those files went away, but perhaps you're right that
> we should only be looking at are things within any cur/ and new/
> subfolders within the message archive.

I don't mean to reduce Notmuch usability to get rid of a few unwanted
messages on stderr. What I suggest is to consider all files outside of
"cur" and "new" to be legitimate non-mail files by default. If Notmuch
chooses to inspect them and detects valid messages, then fine, but the
default assumption should be non-mail.

Think about how the files get there. You manually dumping some *.eml
files in a directory is not exactly typical, IMO. ;-) I expect that in
the majority of cases there is some MTA or fetchmail-like process
storing mail files in a Maildir structure (flat, tree, or mixed like
Dovecot does).

I think that altering Notmuch's expectations about files outside of
well-known subdirectories should be possible without sacrificing its
flexibility.

-Ralph

Re: Unexpected output of "notmuch new --quiet"

[Eric]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 2238 bytes --]

On Sat, 20 Apr 2019 01:46:30 +0200, Ralph Seichter <abbot@monksofcool.net> wrote:
> * Daniel Kahn Gillmor:
> 
> > as far as maildir goes, i think notmuch probably *shouldn't* be
> > looking at stuff in …/tmp/ -- right?
> 
> You're right, I don't see why Notmuch would peek into "tmp".
> 
> > I'd be sad if those files went away, but perhaps you're right that
> > we should only be looking at are things within any cur/ and new/
> > subfolders within the message archive.
> 
> I don't mean to reduce Notmuch usability to get rid of a few unwanted
> messages on stderr. What I suggest is to consider all files outside of
> "cur" and "new" to be legitimate non-mail files by default. If Notmuch
> chooses to inspect them and detects valid messages, then fine, but the
> default assumption should be non-mail.

I suppose that having a configuration item to make notmuch do what it
does now if the default behaviour is changed isn't an enormous usability
reduction, but I still don't like it, and I would be very wary of assuming
that the necessary changes to the code will be as simple as it sounds.

> Think about how the files get there. You manually dumping some *.eml
> files in a directory is not exactly typical, IMO. ;-)

I think that there will be lots of people (and organisations) who get
collections of email from elsewhere and want to add them all to their
own single collection for reference, archival, and research purposes.

> I expect that in the majority of cases there is some MTA or fetchmail-like
> process storing mail files in a Maildir structure (flat, tree, or mixed
> like Dovecot does).

So that those emails would be forced into the IMAP folder structure,
making them visible (and possible distracting) there when viewed with
a traditional mail client?

> I think that altering Notmuch's expectations about files outside of
> well-known subdirectories should be possible without sacrificing its
> flexibility.

No, because that is part of its flexibility. Removing, or degrading,
notmuch's ability to access such files will mean that I may be looking
for a different solution to managing my email. That, of course, amy not
matter much to anyone else, but how many others will there be?

Eric
-- 
ms fnd in a lbry

Re: Unexpected output of "notmuch new --quiet"

[Ralph Seichter]

* Eric:

> I think that there will be lots of people (and organisations) who get
> collections of email from elsewhere and want to add them all to their
> own single collection for reference, archival, and research purposes.

And how would dropping individual mail files into "cur" instead of any
old folder limit that?

> So that those emails would be forced into the IMAP folder structure,
> making them visible (and possible distracting) there when viewed with
> a traditional mail client?

I don't know why you talk about "forced", and it is not an IMAP folder
structure anyway. I have configured Dovecot to use a Maildir store
specifically to allow Notmuch as a client in addition to IMAP. Other
storage types like Multi-dbox would be more efficient for IMAP users.

In my setup, I could circumvent Dovecot's LDA and have Postfix drop off
mail directly in that same Maildir structure, but I want to enable users
to use Sieve scripts. This works because Dovecot and Postfix have very
similar ideas about how a Maildir store should look like.

My point is: Maildir has a defined structure. I'm not against Notmuch
supporting a wider range of layouts, but I think Notmuch should not
complain about perfectly legitimate non-mail files found outside the
Maildir scope.

-Ralph

Re: Unexpected output of "notmuch new --quiet"

[David Bremner]

Ralph Seichter <abbot@monksofcool.net> writes:

> * Eric:
>
>> I think that there will be lots of people (and organisations) who get
>> collections of email from elsewhere and want to add them all to their
>> own single collection for reference, archival, and research purposes.
>
> And how would dropping individual mail files into "cur" instead of any
> old folder limit that?

The issue is more that people have existing mail setups that work, and
we don't want to break them without a good reason; on the other hand
changing the default messages would probably not really count as
breakage.

> My point is: Maildir has a defined structure. I'm not against Notmuch
> supporting a wider range of layouts, but I think Notmuch should not
> complain about perfectly legitimate non-mail files found outside the
> Maildir scope.

Right, I think that point is clear now, that we're just talking about
changing the warning messages. I'm not saying I'm going to work on it,
but I think I understand the feature-request.

d

Re: Unexpected output of "notmuch new --quiet"

[Ralph Seichter]

* David Bremner:

> Right, I think that point is clear now, that we're just talking about
> changing the warning messages.

Indeed. I am actually quite surprised how many reactions there were
about this minor issue. I'm using the new.ignore settings you suggested,
and I can live with that. The warning messages surprised me, is all.

> I'm not saying I'm going to work on it, but I think I understand the
> feature-request.

I'd appreciate a change at some point in the future, time permitting,
but I would not begrudge you spending your energy on more important
issues.

-Ralph