Possible bug with named queries

Possible bug with named queries

[Adam Washington]

I've either stumbled across a bug in the handling of named queries or
(more likely) a bug in their documentation.  I've tried creating some
named queries by adding the following three lines to my
.notmuch-config file.

[query]
sasview=from:sasview.org or (from:github subject:sasview) or (from:github subject:sasmodels)
mantid=from:mantidproject.org

If I run `notmuch config list`, the following lines are amongst the output

crypto.gpg_path=gpg
query.sasview=from:sasview.org or (from:github subject:sasview) or (from:github subject:sasmodels)
query.mantid=from:mantidproject.org
built_with.compact=true

However, the queries do not seem to work.

$ notmuch count from:mantidproject.org
7
$ notmuch count query:mantid
0

Also, while the output from `notmuch config list` showed both queries,
trying to view them gives no result.

$ notmuch config get mantid

$

I'm guessing that the queries aren't configured properly, but I can't
seem to find any documentation that indicates that I'm doing something
wrong.  The man page for notmuch-config states that the explanation
for named queries is in notmuch-search-terms, which, in turn, points
back to notmuch-config.

Is this a bug or have I declared my queries incorrectly?

Re: Possible bug with named queries

[David Bremner]

Adam Washington <rprospero@gmail.com> writes:

> I've either stumbled across a bug in the handling of named queries or
> (more likely) a bug in their documentation.  I've tried creating some
> named queries by adding the following three lines to my
> .notmuch-config file.

you should use "notmuch config" to add named queries. In the latest
version of the docs (will be in the next release), these are noted as
STORED IN DATABASE, thanks to some doc cleanup by dkg. In general I'd
recommend using using "notmuch config" for all configuration changes,
then you don't have to keep track of which ones are stored in
.notmuch-config.  There are technical reasons for certain items being in
the database, and that's likely to continue.

d

Re: Possible bug with named queries

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1109 bytes --]

On Wed, Dec 13 2017, David Bremner wrote:
> STORED IN DATABASE, thanks to some doc cleanup by dkg. In general I'd
> recommend using using "notmuch config" for all configuration changes,
> then you don't have to keep track of which ones are stored in
> .notmuch-config.

Really? We're deprecating the ability to just edit the configuration
file by hand? That's unfortunate.

> There are technical reasons for certain items being in
> the database, and that's likely to continue.

I won't dispute the technical reasons, (I'm out of the loop on that).

But if running "notmuch config" to add a saved search stores some
necessary data in the database, couldn't the same code be invoked when
loading the configuration file and noticing a new saved search there in
the file?

We've gone out of our way to make the configuration file as useful as
possible, (such as by maintaining comments in the file even when using a
tool such as "notmuch config" to edit it). Let's please not throw out
all the benefits by just saying that manually editing the configuration
file is unreliable and people shouldn't do it.

-Carl

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

config file vs. config in the database [was: Re: Possible bug with named queries]

[Daniel Kahn Gillmor]

[-- Attachment #1: Type: text/plain, Size: 6808 bytes --]

Hi Carl, all--

thanks for raising this subject!  I've taken this opportunity to write
down a bit more justification for why i lean in the direction that
Bremner describes, but would be happy to be argued out of it.

On Wed 2017-12-13 15:09:23 -0800, Carl Worth wrote:
> Really? We're deprecating the ability to just edit the configuration
> file by hand? That's unfortunate.

the interaction between the config file and libnotmuch is particularly
problematic -- the config file tends to be something dealt with by the
notmuch cli, but that means that MUAs built on notmuch that *don't* use
the cli as the backend (that is, which use the library directly) don't
get the same advantages.

Also, having two different config files that apply to the same database
presents a not-insignificant logistical challenge -- is it possible to
set configurations that will apply to all accesses that happen to a
given database?  the right place to store those configuration choices is
*in the database itself*.

> But if running "notmuch config" to add a saved search stores some
> necessary data in the database, couldn't the same code be invoked when
> loading the configuration file and noticing a new saved search there in
> the file?

i would support a changeset that does this for the cli, that's a good
idea.  But as mentioned before, it's not clear when this code would be
invoked by a MUA that uses the library directly.  how would you ensure
that they stay in sync?

also, if the database already has a stored query of the same name,
should it be overwritten by the one in the config file?  Should it be
removed from the config file once it has been stored in the database?
if i want to see a list of all my saved searches, where should i look?

> We've gone out of our way to make the configuration file as useful as
> possible, (such as by maintaining comments in the file even when using a
> tool such as "notmuch config" to edit it). Let's please not throw out
> all the benefits by just saying that manually editing the configuration
> file is unreliable and people shouldn't do it.

I actually think that the notmuch config file is far from "as useful as
possible".  :( More concerns with the notmuch configuration file will
follow below.  If i had the time and energy, and no more pressing goals
for notmuch, i would probably submit a patch that removes every option
from the configuration file except for the location of the database
itself.  and i'd warn users of the cli about the presence of any other
configuration that remains in their personal configuration file.

I'd even like to see notmuch usable without a configuration file at all,
by selecting a default location for the database path (though i don't
have the political stamina today to fight the inevitable fight about
where that path would be).

I'm happy to hear pushback on this mini-agenda of mine, to better
understand why people want to edit and read this file by hand, but i
currently see the config file as the cause of several minor
papercuts/annoyances, and not a lot of ways that it helps.


writing defaults to the config file
-----------------------------------

the current notmuch configuration subsystem actively writes the defaults
into the config file upon first initializing it means that there is no
way for a user to say "keep this setting as its default".

Consider a simple boolean setting, which defaults initially to false.
Anyone who sets up a config file gets it written down as false.

over time, the project evolves and the new default of the setting is
true.  but everyone who moves to the new version from the old one
retains "false" for that config setting, *even if they have never
explicitly set it to false*, because notmuch (rightly) doesn't want to
change the values in the file.


writing standard comments to the config file
--------------------------------------------

by default, when notmuch creates an initial config file, it stuffs a
bunch of comments about what goes there in it, in addition to including
the initial settings themselves.

My experience with heavily-commented, detailed, per-user configuration
files is that they are great ways to accumulate cruft over time.  For an
installation of complex software that has existed over years, any user's
configuration file that contains auto-generated comments or auto-written
values likely contains choices or information that are *factually wrong*
for the current version of the software that they're using.

This is the reason i pushed GnuPG into getting rid of their default
configuration file.  Modern versions of gpg do not create a gpg.conf at
all, and (usually) Just Work™ without anything there.  Each installation
still ships documentation about what options *can* go in the config
file, though.

Notmuch has a good documentation system, and the configuration options
for the currently-installed version of notmuch are well-documented in
notmuch-config(1), along with their defaults.  We shouldn't be using the
user's configuration file as a place to stash software documentation.


using the human-readable config file for "safe backup"
------------------------------------------------------

One advantage of the human-readable config file is that it can provide a
backup format, no matter how xapian or notmuch changes in the future.  a
plain text serialization/deserialization format is a handy thing to
have.

however, we already have that with "notmuch dump" and "notmuch restore",
and it's not clear that the config file offers any advantage over those
subcommands for the purposes of backup and restore.

arguably, the config file is marginally worse, because it means that you
have to back up two things instead of of one if you want a recoverable
text-based backup.



using the config file for the user's notes about notmuch configuration
----------------------------------------------------------------------

the one thing my proposal above doesn't cover is a place for users to
put notes about why they've chosen a particular configuration option,
which they can currently do in comments in the config file.

I don't actually have a proposal for how to deal with this, other than
to observe that if there was no config file, users could still take
notes anywhere (including wherever it is that they take notes about how
they choose to use their web browser, their word processor, their
compiler, etc).  If someone wants to propose an improvement here, i'm
all ears.

Do people want to offer examples of user-written notes that they have as
comments in their config file that they would be sad about losing if the
config file were to go away?

All the best,

       --dkg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Possible bug with named queries

[David Bremner]

Carl Worth <cworth@cworth.org> writes:

> But if running "notmuch config" to add a saved search stores some
> necessary data in the database, couldn't the same code be invoked when
> loading the configuration file and noticing a new saved search there in
> the file?

It could (with some complications, e.g. for users of libnotmuch). But
the version in the database can change via the libnotmuch API, which
presumably should be written to the config file. I'm not sure how that
would all work.

> We've gone out of our way to make the configuration file as useful as
> possible, (such as by maintaining comments in the file even when using a
> tool such as "notmuch config" to edit it). Let's please not throw out
> all the benefits by just saying that manually editing the configuration
> file is unreliable and people shouldn't do it.

Well, it's reliable if people read the (new) documentation
carefully. But obviously it's a bit surprising that some things are just
ignored in .notmuch-config. It should be possible to report warnings for
those items.

IIRC, one thing we (collectively) discussed and rejected is having a
seperate command to manipulate database resident configuration
information. That could still be done.

d