* [Patch v2 01/13] configure: autodetect xapian-1.3
2016-03-26 17:57 v2 of libconfig, date:foo, and named query patches David Bremner
@ 2016-03-26 17:57 ` David Bremner
2016-04-01 9:09 ` Tomi Ollila
2016-04-01 23:29 ` David Bremner
2016-03-26 17:57 ` [Patch v2 02/13] configure: detect Xapian:FieldProcessor David Bremner
` (12 subsequent siblings)
13 siblings, 2 replies; 38+ messages in thread
From: David Bremner @ 2016-03-26 17:57 UTC (permalink / raw)
To: notmuch
Mimic the handling of python2 versus python3. In particular if both
xapian-config and xapian-config-1.3 are found, use xapian-config
---
configure | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/configure b/configure
index a79f6bd..eb6dbac 100755
--- a/configure
+++ b/configure
@@ -51,7 +51,7 @@ CPPFLAGS=${CPPFLAGS:-}
CXXFLAGS_for_sh=${CXXFLAGS:-${CFLAGS}}
CXXFLAGS=${CXXFLAGS:-\$(CFLAGS)}
LDFLAGS=${LDFLAGS:-}
-XAPIAN_CONFIG=${XAPIAN_CONFIG:-xapian-config}
+XAPIAN_CONFIG=${XAPIAN_CONFIG:-}
PYTHON=${PYTHON:-}
# We don't allow the EMACS or GZIP Makefile variables inherit values
@@ -341,7 +341,7 @@ fi
printf "Checking for Xapian development files... "
have_xapian=0
-for xapian_config in ${XAPIAN_CONFIG}; do
+for xapian_config in ${XAPIAN_CONFIG} xapian-config xapian-config-1.3; do
if ${xapian_config} --version > /dev/null 2>&1; then
xapian_version=$(${xapian_config} --version | sed -e 's/.* //')
printf "Yes (%s).\n" ${xapian_version}
--
2.6.4
^ permalink raw reply related [flat|nested] 38+ messages in thread* Re: [Patch v2 01/13] configure: autodetect xapian-1.3
2016-03-26 17:57 ` [Patch v2 01/13] configure: autodetect xapian-1.3 David Bremner
@ 2016-04-01 9:09 ` Tomi Ollila
2016-04-01 23:29 ` David Bremner
1 sibling, 0 replies; 38+ messages in thread
From: Tomi Ollila @ 2016-04-01 9:09 UTC (permalink / raw)
To: David Bremner, notmuch
On Sat, Mar 26 2016, David Bremner <david@tethera.net> wrote:
> Mimic the handling of python2 versus python3. In particular if both
> xapian-config and xapian-config-1.3 are found, use xapian-config
+1 (this change)
> ---
> configure | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/configure b/configure
> index a79f6bd..eb6dbac 100755
> --- a/configure
> +++ b/configure
> @@ -51,7 +51,7 @@ CPPFLAGS=${CPPFLAGS:-}
> CXXFLAGS_for_sh=${CXXFLAGS:-${CFLAGS}}
> CXXFLAGS=${CXXFLAGS:-\$(CFLAGS)}
> LDFLAGS=${LDFLAGS:-}
> -XAPIAN_CONFIG=${XAPIAN_CONFIG:-xapian-config}
> +XAPIAN_CONFIG=${XAPIAN_CONFIG:-}
> PYTHON=${PYTHON:-}
>
> # We don't allow the EMACS or GZIP Makefile variables inherit values
> @@ -341,7 +341,7 @@ fi
>
> printf "Checking for Xapian development files... "
> have_xapian=0
> -for xapian_config in ${XAPIAN_CONFIG}; do
> +for xapian_config in ${XAPIAN_CONFIG} xapian-config xapian-config-1.3; do
> if ${xapian_config} --version > /dev/null 2>&1; then
> xapian_version=$(${xapian_config} --version | sed -e 's/.* //')
> printf "Yes (%s).\n" ${xapian_version}
> --
> 2.6.4
^ permalink raw reply [flat|nested] 38+ messages in thread* Re: [Patch v2 01/13] configure: autodetect xapian-1.3
2016-03-26 17:57 ` [Patch v2 01/13] configure: autodetect xapian-1.3 David Bremner
2016-04-01 9:09 ` Tomi Ollila
@ 2016-04-01 23:29 ` David Bremner
1 sibling, 0 replies; 38+ messages in thread
From: David Bremner @ 2016-04-01 23:29 UTC (permalink / raw)
To: notmuch
David Bremner <david@tethera.net> writes:
> Mimic the handling of python2 versus python3. In particular if both
> xapian-config and xapian-config-1.3 are found, use xapian-config
pushed this one patch
^ permalink raw reply [flat|nested] 38+ messages in thread
* [Patch v2 02/13] configure: detect Xapian:FieldProcessor
2016-03-26 17:57 v2 of libconfig, date:foo, and named query patches David Bremner
2016-03-26 17:57 ` [Patch v2 01/13] configure: autodetect xapian-1.3 David Bremner
@ 2016-03-26 17:57 ` David Bremner
2016-03-26 17:57 ` [Patch v2 03/13] lib: optionally support single argument date: queries David Bremner
` (11 subsequent siblings)
13 siblings, 0 replies; 38+ messages in thread
From: David Bremner @ 2016-03-26 17:57 UTC (permalink / raw)
To: notmuch
Rather than check versions, it seems more robust to do a test compile.
---
configure | 26 +++++++++++++++++++++++++-
1 file changed, 25 insertions(+), 1 deletion(-)
diff --git a/configure b/configure
index eb6dbac..c48f7ce 100755
--- a/configure
+++ b/configure
@@ -356,9 +356,10 @@ if [ ${have_xapian} = "0" ]; then
errors=$((errors + 1))
fi
-# Compaction is only supported on Xapian > 1.2.6
have_xapian_compact=0
+have_xapian_field_processor=0
if [ ${have_xapian} = "1" ]; then
+ # Compaction is only supported on Xapian > 1.2.6
printf "Checking for Xapian compaction support... "
case "${xapian_version}" in
0.*|1.[01].*|1.2.[0-5])
@@ -369,6 +370,21 @@ if [ ${have_xapian} = "1" ]; then
*)
printf "Unknown version.\n" ;;
esac
+
+ printf "Checking for Xapian FieldProcessor API... "
+ cat>_field_processor.cc<<EOF
+#include <xapian.h>
+class TitleFieldProcessor : public Xapian::FieldProcessor { };
+EOF
+ if ${CXX} ${CXXLAGS} ${xapian_cxxflags} -c _field_processor.cc -o _field_processor.o > /dev/null 2>&1
+ then
+ have_xapian_field_processor=1
+ printf "Yes.\n"
+ else
+ printf "No. (optional)\n"
+ fi
+
+ rm -f _field_processor.o _field_processor.cc
fi
@@ -979,6 +995,9 @@ HAVE_D_TYPE = ${have_d_type}
# Whether the Xapian version in use supports compaction
HAVE_XAPIAN_COMPACT = ${have_xapian_compact}
+# Whether the Xapian version in use supports field processors
+HAVE_XAPIAN_FIELD_PROCESSOR = ${have_xapian_field_processor}
+
# Whether the getpwuid_r function is standards-compliant
# (if not, then notmuch will #define _POSIX_PTHREAD_SEMANTICS
# to enable the standards-compliant version -- needed for Solaris)
@@ -1050,6 +1069,7 @@ CONFIGURE_CFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) \$(GMIME_CFLAGS) \\
-DSTD_GETPWUID=\$(STD_GETPWUID) \\
-DSTD_ASCTIME=\$(STD_ASCTIME) \\
-DHAVE_XAPIAN_COMPACT=\$(HAVE_XAPIAN_COMPACT) \\
+ -DHAVE_XAPIAN_FIELD_PROCESSOR=\$(HAVE_XAPIAN_PROCESSOR) \\
-DUTIL_BYTE_ORDER=\$(UTIL_BYTE_ORDER)
CONFIGURE_CXXFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) \$(GMIME_CFLAGS) \\
@@ -1064,6 +1084,7 @@ CONFIGURE_CXXFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) \$(GMIME_CFLAGS) \\
-DSTD_GETPWUID=\$(STD_GETPWUID) \\
-DSTD_ASCTIME=\$(STD_ASCTIME) \\
-DHAVE_XAPIAN_COMPACT=\$(HAVE_XAPIAN_COMPACT) \\
+ -DHAVE_XAPIAN_FIELD_PROCESSOR=\$(HAVE_XAPIAN_FIELD_PROCESSOR) \\
-DUTIL_BYTE_ORDER=\$(UTIL_BYTE_ORDER)
CONFIGURE_LDFLAGS = \$(GMIME_LDFLAGS) \$(TALLOC_LDFLAGS) \$(ZLIB_LDFLAGS) \$(XAPIAN_LDFLAGS)
@@ -1077,6 +1098,9 @@ cat > sh.config <<EOF
# Whether the Xapian version in use supports compaction
NOTMUCH_HAVE_XAPIAN_COMPACT=${have_xapian_compact}
+# Whether the Xapian version in use supports field processors
+NOTMUCH_HAVE_XAPIAN_FIELD_PROCESSOR=${have_xapian_field_processor}
+
# do we have man pages?
NOTMUCH_HAVE_MAN=$((have_sphinx))
--
2.6.4
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Patch v2 03/13] lib: optionally support single argument date: queries
2016-03-26 17:57 v2 of libconfig, date:foo, and named query patches David Bremner
2016-03-26 17:57 ` [Patch v2 01/13] configure: autodetect xapian-1.3 David Bremner
2016-03-26 17:57 ` [Patch v2 02/13] configure: detect Xapian:FieldProcessor David Bremner
@ 2016-03-26 17:57 ` David Bremner
2016-03-26 17:57 ` [Patch v2 04/13] lib/cli: add library API / CLI for compile time options David Bremner
` (10 subsequent siblings)
13 siblings, 0 replies; 38+ messages in thread
From: David Bremner @ 2016-03-26 17:57 UTC (permalink / raw)
To: notmuch
This relies on the FieldProcessor API, which is only present in xapian
>= 1.3.
---
doc/man7/notmuch-search-terms.rst | 22 +++++++++++++++++++---
lib/database-private.h | 3 +++
lib/database.cc | 6 ++++++
lib/parse-time-vrp.cc | 21 +++++++++++++++++++++
lib/parse-time-vrp.h | 5 +++++
test/T500-search-date.sh | 6 ++++++
6 files changed, 60 insertions(+), 3 deletions(-)
diff --git a/doc/man7/notmuch-search-terms.rst b/doc/man7/notmuch-search-terms.rst
index 2fbc16d..29e8b03 100644
--- a/doc/man7/notmuch-search-terms.rst
+++ b/doc/man7/notmuch-search-terms.rst
@@ -281,9 +281,10 @@ matches from the beginning of January to the end of February.
date:<expr>..! can be used as a shorthand for date:<expr>..<expr>. The
expansion takes place before interpretation, and thus, for example,
date:monday..! matches from the beginning of Monday until the end of
-Monday. (Note that entering date:<expr> without "..", for example
-date:yesterday, won't work, as it's not interpreted as a range
-expression at all. Again, use date:yesterday..!)
+Monday.
+With **Xapian Field Processor** support (see below), non-range
+date queries such as date:yesterday will work, but otherwise
+will give unexpected results; if in doubt use date:yesterday..!
Currently, we do not support spaces in range expressions. You can
replace the spaces with '\_', or (in most cases) '-', or (in some cases)
@@ -370,6 +371,21 @@ Time zones
Some time zone codes, e.g. UTC, EET.
+XAPIAN FIELD PROCESSORS
+=======================
+
+Certain optional features of the notmuch query processor rely on the
+presence of the Xapian field processor API. You can determine if your
+notmuch was built against a sufficiently recent version of Xapian by running
+
+::
+
+ % notmuch config get options.field_processor
+
+Currently the following features require field processor support:
+
+- non-range date queries, e.g. "date:today"
+
SEE ALSO
========
diff --git a/lib/database-private.h b/lib/database-private.h
index 3fb10f7..e1962f4 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -176,6 +176,9 @@ struct _notmuch_database {
Xapian::TermGenerator *term_gen;
Xapian::ValueRangeProcessor *value_range_processor;
Xapian::ValueRangeProcessor *date_range_processor;
+#if HAVE_XAPIAN_FIELD_PROCESSOR
+ Xapian::FieldProcessor *date_field_processor;
+#endif
Xapian::ValueRangeProcessor *last_mod_range_processor;
};
diff --git a/lib/database.cc b/lib/database.cc
index 3b342f1..0e3c90c 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1000,6 +1000,12 @@ notmuch_database_open_verbose (const char *path,
notmuch->term_gen->set_stemmer (Xapian::Stem ("english"));
notmuch->value_range_processor = new Xapian::NumberValueRangeProcessor (NOTMUCH_VALUE_TIMESTAMP);
notmuch->date_range_processor = new ParseTimeValueRangeProcessor (NOTMUCH_VALUE_TIMESTAMP);
+#if HAVE_XAPIAN_FIELD_PROCESSOR
+ /* This currently relies on the query parser to pass anything
+ * with a .. to the range processor */
+ notmuch->date_field_processor = new DateFieldProcessor();
+ notmuch->query_parser->add_boolean_prefix("date", notmuch->date_field_processor);
+#endif
notmuch->last_mod_range_processor = new Xapian::NumberValueRangeProcessor (NOTMUCH_VALUE_LAST_MOD, "lastmod:");
notmuch->query_parser->set_default_op (Xapian::Query::OP_AND);
diff --git a/lib/parse-time-vrp.cc b/lib/parse-time-vrp.cc
index 03804cf..b15b77c 100644
--- a/lib/parse-time-vrp.cc
+++ b/lib/parse-time-vrp.cc
@@ -64,3 +64,24 @@ ParseTimeValueRangeProcessor::operator() (std::string &begin, std::string &end)
return valno;
}
+
+#if HAVE_XAPIAN_FIELD_PROCESSOR
+/* XXX TODO: is throwing an exception the right thing to do here? */
+Xapian::Query DateFieldProcessor::operator()(const std::string & str) {
+ time_t from, to, now;
+
+ /* Use the same 'now' for begin and end. */
+ if (time (&now) == (time_t) -1)
+ throw Xapian::QueryParserError("Unable to get current time");
+
+ if (parse_time_string (str.c_str (), &from, &now, PARSE_TIME_ROUND_DOWN))
+ throw Xapian::QueryParserError ("Didn't understand date specification '" + str + "'");
+
+ if (parse_time_string (str.c_str (), &to, &now, PARSE_TIME_ROUND_UP_INCLUSIVE))
+ throw Xapian::QueryParserError ("Didn't understand date specification '" + str + "'");
+
+ return Xapian::Query(Xapian::Query::OP_AND,
+ Xapian::Query(Xapian::Query::OP_VALUE_GE, 0, Xapian::sortable_serialise ((double) from)),
+ Xapian::Query(Xapian::Query::OP_VALUE_LE, 0, Xapian::sortable_serialise ((double) to)));
+}
+#endif
diff --git a/lib/parse-time-vrp.h b/lib/parse-time-vrp.h
index 094c4f8..3bd12bf 100644
--- a/lib/parse-time-vrp.h
+++ b/lib/parse-time-vrp.h
@@ -37,4 +37,9 @@ public:
Xapian::valueno operator() (std::string &begin, std::string &end);
};
+#if HAVE_XAPIAN_FIELD_PROCESSOR
+class DateFieldProcessor : public Xapian::FieldProcessor {
+ Xapian::Query operator()(const std::string & str);
+};
+#endif
#endif /* NOTMUCH_PARSE_TIME_VRP_H */
diff --git a/test/T500-search-date.sh b/test/T500-search-date.sh
index f5cea42..198a2e6 100755
--- a/test/T500-search-date.sh
+++ b/test/T500-search-date.sh
@@ -12,6 +12,12 @@ test_begin_subtest "Absolute date range with 'same' operator"
output=$(notmuch search date:2010-12-16..! | notmuch_search_sanitize)
test_expect_equal "$output" "thread:XXX 2010-12-16 [1/1] Olivier Berger; Essai accentué (inbox unread)"
+if [ "${NOTMUCH_HAVE_XAPIAN_FIELD_PROCESSOR}" = "1" ]; then
+ test_begin_subtest "Absolute date field"
+ output=$(notmuch search date:2010-12-16 | notmuch_search_sanitize)
+ test_expect_equal "$output" "thread:XXX 2010-12-16 [1/1] Olivier Berger; Essai accentué (inbox unread)"
+fi
+
test_begin_subtest "Absolute time range with TZ"
notmuch search date:18-Nov-2009_02:19:26-0800..2009-11-18_04:49:52-06:00 | notmuch_search_sanitize > OUTPUT
cat <<EOF >EXPECTED
--
2.6.4
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Patch v2 04/13] lib/cli: add library API / CLI for compile time options
2016-03-26 17:57 v2 of libconfig, date:foo, and named query patches David Bremner
` (2 preceding siblings ...)
2016-03-26 17:57 ` [Patch v2 03/13] lib: optionally support single argument date: queries David Bremner
@ 2016-03-26 17:57 ` David Bremner
2016-04-27 17:47 ` Tomi Ollila
2016-03-26 17:57 ` [Patch v2 05/13] configure: check directly for xapian compaction API David Bremner
` (9 subsequent siblings)
13 siblings, 1 reply; 38+ messages in thread
From: David Bremner @ 2016-03-26 17:57 UTC (permalink / raw)
To: notmuch
This is intentionally low tech; if we have more than two options it may
make sense to build up what infrastructure is provided.
---
doc/man1/notmuch-config.rst | 5 +++++
lib/Makefile.local | 1 +
lib/notmuch.h | 10 +++++++++
lib/options.c | 50 +++++++++++++++++++++++++++++++++++++++++++++
notmuch-config.c | 20 ++++++++++++++++++
test/T030-config.sh | 6 ++++--
test/T040-setup.sh | 6 ++++--
test/test-lib.sh | 6 ++++++
8 files changed, 100 insertions(+), 4 deletions(-)
create mode 100644 lib/options.c
diff --git a/doc/man1/notmuch-config.rst b/doc/man1/notmuch-config.rst
index 40c1272..150d764 100644
--- a/doc/man1/notmuch-config.rst
+++ b/doc/man1/notmuch-config.rst
@@ -132,6 +132,11 @@ The available configuration items are described below.
Default: ``gpg``.
+ **options.<name>**
+
+ Compile time option <name>. Current possibilities include
+ "compact" (see **notmuch-compact(1)**)
+ and "field_processor" (see **notmuch-search-terms(7)**).
ENVIRONMENT
===========
diff --git a/lib/Makefile.local b/lib/Makefile.local
index 3a07090..4ad0158 100644
--- a/lib/Makefile.local
+++ b/lib/Makefile.local
@@ -39,6 +39,7 @@ libnotmuch_c_srcs = \
$(dir)/message-file.c \
$(dir)/messages.c \
$(dir)/sha1.c \
+ $(dir)/options.c \
$(dir)/tags.c
libnotmuch_cxx_srcs = \
diff --git a/lib/notmuch.h b/lib/notmuch.h
index cb46fc0..b29dd5f 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -1838,6 +1838,16 @@ notmuch_filenames_move_to_next (notmuch_filenames_t *filenames);
void
notmuch_filenames_destroy (notmuch_filenames_t *filenames);
+typedef enum {
+ NOTMUCH_OPTION_COMPACT = 1,
+ NOTMUCH_OPTION_FIELD_PROCESSOR = 2
+} notmuch_option_t;
+
+notmuch_bool_t
+notmuch_options_present (notmuch_option_t mask);
+
+notmuch_bool_t
+notmuch_options_get (const char *name);
/* @} */
NOTMUCH_END_DECLS
diff --git a/lib/options.c b/lib/options.c
new file mode 100644
index 0000000..4e15d92
--- /dev/null
+++ b/lib/options.c
@@ -0,0 +1,50 @@
+/* notmuch - Not much of an email program, (just index and search)
+ *
+ * Copyright © 2016 David Bremner
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see http://www.gnu.org/licenses/ .
+ *
+ * Author: David Bremner <david@tethera.net>
+ */
+
+#include "notmuch.h"
+#include "notmuch-private.h"
+
+notmuch_bool_t
+notmuch_options_present (notmuch_option_t mask)
+{
+ notmuch_option_t present = 0;
+
+#if HAVE_XAPIAN_COMPACT
+ present |= NOTMUCH_OPTION_COMPACT;
+#endif
+
+#if HAVE_XAPIAN_COMPACT
+ present |= NOTMUCH_OPTION_FIELD_PROCESSOR;
+#endif
+
+ return (mask & present) != 0;
+
+}
+
+notmuch_bool_t
+notmuch_options_get (const char *name) {
+ if (STRNCMP_LITERAL (name, "compact") == 0) {
+ return notmuch_options_present (NOTMUCH_OPTION_COMPACT);
+ } else if (STRNCMP_LITERAL (name, "field_processor") == 0) {
+ return notmuch_options_present (NOTMUCH_OPTION_FIELD_PROCESSOR);
+ } else {
+ return FALSE;
+ }
+}
diff --git a/notmuch-config.c b/notmuch-config.c
index d252bb2..cfc549d 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -750,6 +750,8 @@ _item_split (char *item, char **group, char **key)
return 0;
}
+#define OPTION_PREFIX "options."
+
static int
notmuch_config_command_get (notmuch_config_t *config, char *item)
{
@@ -773,6 +775,9 @@ notmuch_config_command_get (notmuch_config_t *config, char *item)
tags = notmuch_config_get_new_tags (config, &length);
for (i = 0; i < length; i++)
printf ("%s\n", tags[i]);
+ } else if (STRNCMP_LITERAL (item, OPTION_PREFIX) == 0) {
+ printf ("%s\n",
+ notmuch_options_get (item + strlen (OPTION_PREFIX)) ? "true" : "false");
} else {
char **value;
size_t i, length;
@@ -804,6 +809,11 @@ notmuch_config_command_set (notmuch_config_t *config, char *item, int argc, char
{
char *group, *key;
+ if (STRNCMP_LITERAL (item, OPTION_PREFIX) == 0) {
+ fprintf (stderr, "Error: read only option: %s\n", item);
+ return 1;
+ }
+
if (_item_split (item, &group, &key))
return 1;
@@ -830,6 +840,15 @@ notmuch_config_command_set (notmuch_config_t *config, char *item, int argc, char
return notmuch_config_save (config);
}
+static
+void
+_notmuch_config_list_options () {
+ printf("options.compact=%s\n",
+ notmuch_options_present(NOTMUCH_OPTION_COMPACT) ? "true" : "false");
+ printf("options.field_processor=%s\n",
+ notmuch_options_present(NOTMUCH_OPTION_FIELD_PROCESSOR) ? "true" : "false");
+}
+
static int
notmuch_config_command_list (notmuch_config_t *config)
{
@@ -865,6 +884,7 @@ notmuch_config_command_list (notmuch_config_t *config)
g_strfreev (groups);
+ _notmuch_config_list_options ();
return 0;
}
diff --git a/test/T030-config.sh b/test/T030-config.sh
index f404908..c37ba21 100755
--- a/test/T030-config.sh
+++ b/test/T030-config.sh
@@ -44,7 +44,7 @@ test_expect_equal "$(notmuch config get foo.nonexistent)" ""
test_begin_subtest "List all items"
notmuch config set database.path "/canonical/path"
-output=$(notmuch config list)
+output=$(notmuch config list | notmuch_options_sanitize)
test_expect_equal "$output" "\
database.path=/canonical/path
user.name=Notmuch Test Suite
@@ -56,7 +56,9 @@ search.exclude_tags=
maildir.synchronize_flags=true
crypto.gpg_path=gpg
foo.string=this is another string value
-foo.list=this;is another;list value;"
+foo.list=this;is another;list value;
+options.compact=something
+options.field_processor=something"
test_begin_subtest "Top level --config=FILE option"
cp "${NOTMUCH_CONFIG}" alt-config
diff --git a/test/T040-setup.sh b/test/T040-setup.sh
index cf0c00b..5db03a6 100755
--- a/test/T040-setup.sh
+++ b/test/T040-setup.sh
@@ -19,7 +19,7 @@ another.suite@example.com
foo bar
baz
EOF
-output=$(notmuch --config=new-notmuch-config config list)
+output=$(notmuch --config=new-notmuch-config config list | notmuch_options_sanitize)
test_expect_equal "$output" "\
database.path=/path/to/maildir
user.name=Test Suite
@@ -29,6 +29,8 @@ new.tags=foo;bar;
new.ignore=
search.exclude_tags=baz;
maildir.synchronize_flags=true
-crypto.gpg_path=gpg"
+crypto.gpg_path=gpg
+options.compact=something
+options.field_processor=something"
test_done
diff --git a/test/test-lib.sh b/test/test-lib.sh
index cc08a98..49e6f40 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -733,6 +733,12 @@ notmuch_uuid_sanitize ()
{
sed 's/[0-9a-f]\{8\}-[0-9a-f]\{4\}-[0-9a-f]\{4\}-[0-9a-f]\{4\}-[0-9a-f]\{12\}/UUID/g'
}
+
+notmuch_options_sanitize ()
+{
+ sed 's/^options[.]\(.*\)=.*$/options.\1=something/'
+}
+
# End of notmuch helper functions
# Use test_set_prereq to tell that a particular prerequisite is available.
--
2.6.4
^ permalink raw reply related [flat|nested] 38+ messages in thread* Re: [Patch v2 04/13] lib/cli: add library API / CLI for compile time options
2016-03-26 17:57 ` [Patch v2 04/13] lib/cli: add library API / CLI for compile time options David Bremner
@ 2016-04-27 17:47 ` Tomi Ollila
2016-04-30 11:53 ` David Bremner
0 siblings, 1 reply; 38+ messages in thread
From: Tomi Ollila @ 2016-04-27 17:47 UTC (permalink / raw)
To: David Bremner, notmuch
On Sat, Mar 26 2016, David Bremner <david@tethera.net> wrote:
> This is intentionally low tech; if we have more than two options it may
> make sense to build up what infrastructure is provided.
2 quick notes for this patch -- since it has been like a month when I
looked this it takes time to get re-acquainted to the series again
...IMO this compile time "options" feels a bit odd; what options those are
if those are not options (but fixed things on what is available on the
underlying libraries). something like "features" would be a little less
odd (if there were no "database features" there ;/)
another thing is that these fixed things that were resolved by compile time
is (re-)checked in compile time. instead of that just the
HAVE_XAPIAN_COMPACT and HAVE_XAPIAN_FIELD_PROCESSOR could be used
... like:
notmuch_bool_t
notmuch_options_get (const char *name) {
if (STRNCMP_LITERAL (name, "compact") == 0) {
return HAVE_XAPIAN_COMPACT;
// or return HAVE_XAPIAN_COMPACT? TRUE: FALSE;
// or return !! HAVE_XAPIAN_COMPACT;
} else if (STRNCMP_LITERAL (name, "field_processor") == 0) {
return HAVE_XAPIAN_FIELD_PROCESSOR;
} else {
return FALSE;
}
}
... note: if this notmuch_options_present () function prevails, fix
the later #if check ;D
... and
void
_notmuch_config_list_options () {
printf("options.compact=%s\n",
HAVE_XAPIAN_COMPACT ? "true" : "false");
printf("options.field_processor=%s\n",
HAVE_XAPIAN_FIELD_PROCESSOR ? "true" : "false");
}
(kept the 'options' naming and prefix in this context)
otoh, all this "options" reading using notmuch config interface could
be pos^H^H^H^H put into another series and this naming thing could be
resolved there (proviced that these #if HAVE_XAPIAN ... #endif constructs
can be used conveniently here)
this is how these things look to me now; as usual i may not have
caught all of the considerations here.
Tomi
> ---
> doc/man1/notmuch-config.rst | 5 +++++
> lib/Makefile.local | 1 +
> lib/notmuch.h | 10 +++++++++
> lib/options.c | 50 +++++++++++++++++++++++++++++++++++++++++++++
> notmuch-config.c | 20 ++++++++++++++++++
> test/T030-config.sh | 6 ++++--
> test/T040-setup.sh | 6 ++++--
> test/test-lib.sh | 6 ++++++
> 8 files changed, 100 insertions(+), 4 deletions(-)
> create mode 100644 lib/options.c
>
> diff --git a/doc/man1/notmuch-config.rst b/doc/man1/notmuch-config.rst
> index 40c1272..150d764 100644
> --- a/doc/man1/notmuch-config.rst
> +++ b/doc/man1/notmuch-config.rst
> @@ -132,6 +132,11 @@ The available configuration items are described below.
>
> Default: ``gpg``.
>
> + **options.<name>**
> +
> + Compile time option <name>. Current possibilities include
> + "compact" (see **notmuch-compact(1)**)
> + and "field_processor" (see **notmuch-search-terms(7)**).
>
> ENVIRONMENT
> ===========
> diff --git a/lib/Makefile.local b/lib/Makefile.local
> index 3a07090..4ad0158 100644
> --- a/lib/Makefile.local
> +++ b/lib/Makefile.local
> @@ -39,6 +39,7 @@ libnotmuch_c_srcs = \
> $(dir)/message-file.c \
> $(dir)/messages.c \
> $(dir)/sha1.c \
> + $(dir)/options.c \
> $(dir)/tags.c
>
> libnotmuch_cxx_srcs = \
> diff --git a/lib/notmuch.h b/lib/notmuch.h
> index cb46fc0..b29dd5f 100644
> --- a/lib/notmuch.h
> +++ b/lib/notmuch.h
> @@ -1838,6 +1838,16 @@ notmuch_filenames_move_to_next (notmuch_filenames_t *filenames);
> void
> notmuch_filenames_destroy (notmuch_filenames_t *filenames);
>
> +typedef enum {
> + NOTMUCH_OPTION_COMPACT = 1,
> + NOTMUCH_OPTION_FIELD_PROCESSOR = 2
> +} notmuch_option_t;
> +
> +notmuch_bool_t
> +notmuch_options_present (notmuch_option_t mask);
> +
> +notmuch_bool_t
> +notmuch_options_get (const char *name);
> /* @} */
>
> NOTMUCH_END_DECLS
> diff --git a/lib/options.c b/lib/options.c
> new file mode 100644
> index 0000000..4e15d92
> --- /dev/null
> +++ b/lib/options.c
> @@ -0,0 +1,50 @@
> +/* notmuch - Not much of an email program, (just index and search)
> + *
> + * Copyright © 2016 David Bremner
> + *
> + * This program is free software: you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation, either version 3 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program. If not, see http://www.gnu.org/licenses/ .
> + *
> + * Author: David Bremner <david@tethera.net>
> + */
> +
> +#include "notmuch.h"
> +#include "notmuch-private.h"
> +
> +notmuch_bool_t
> +notmuch_options_present (notmuch_option_t mask)
> +{
> + notmuch_option_t present = 0;
> +
> +#if HAVE_XAPIAN_COMPACT
> + present |= NOTMUCH_OPTION_COMPACT;
> +#endif
> +
> +#if HAVE_XAPIAN_COMPACT
> + present |= NOTMUCH_OPTION_FIELD_PROCESSOR;
> +#endif
> +
> + return (mask & present) != 0;
> +
> +}
> +
> +notmuch_bool_t
> +notmuch_options_get (const char *name) {
> + if (STRNCMP_LITERAL (name, "compact") == 0) {
> + return notmuch_options_present (NOTMUCH_OPTION_COMPACT);
> + } else if (STRNCMP_LITERAL (name, "field_processor") == 0) {
> + return notmuch_options_present (NOTMUCH_OPTION_FIELD_PROCESSOR);
> + } else {
> + return FALSE;
> + }
> +}
> diff --git a/notmuch-config.c b/notmuch-config.c
> index d252bb2..cfc549d 100644
> --- a/notmuch-config.c
> +++ b/notmuch-config.c
> @@ -750,6 +750,8 @@ _item_split (char *item, char **group, char **key)
> return 0;
> }
>
> +#define OPTION_PREFIX "options."
> +
> static int
> notmuch_config_command_get (notmuch_config_t *config, char *item)
> {
> @@ -773,6 +775,9 @@ notmuch_config_command_get (notmuch_config_t *config, char *item)
> tags = notmuch_config_get_new_tags (config, &length);
> for (i = 0; i < length; i++)
> printf ("%s\n", tags[i]);
> + } else if (STRNCMP_LITERAL (item, OPTION_PREFIX) == 0) {
> + printf ("%s\n",
> + notmuch_options_get (item + strlen (OPTION_PREFIX)) ? "true" : "false");
> } else {
> char **value;
> size_t i, length;
> @@ -804,6 +809,11 @@ notmuch_config_command_set (notmuch_config_t *config, char *item, int argc, char
> {
> char *group, *key;
>
> + if (STRNCMP_LITERAL (item, OPTION_PREFIX) == 0) {
> + fprintf (stderr, "Error: read only option: %s\n", item);
> + return 1;
> + }
> +
> if (_item_split (item, &group, &key))
> return 1;
>
> @@ -830,6 +840,15 @@ notmuch_config_command_set (notmuch_config_t *config, char *item, int argc, char
> return notmuch_config_save (config);
> }
>
> +static
> +void
> +_notmuch_config_list_options () {
> + printf("options.compact=%s\n",
> + notmuch_options_present(NOTMUCH_OPTION_COMPACT) ? "true" : "false");
> + printf("options.field_processor=%s\n",
> + notmuch_options_present(NOTMUCH_OPTION_FIELD_PROCESSOR) ? "true" : "false");
> +}
^ permalink raw reply [flat|nested] 38+ messages in thread* Re: [Patch v2 04/13] lib/cli: add library API / CLI for compile time options
2016-04-27 17:47 ` Tomi Ollila
@ 2016-04-30 11:53 ` David Bremner
2016-05-01 17:48 ` Tomi Ollila
0 siblings, 1 reply; 38+ messages in thread
From: David Bremner @ 2016-04-30 11:53 UTC (permalink / raw)
To: Tomi Ollila, notmuch
Tomi Ollila <tomi.ollila@iki.fi> writes:
> On Sat, Mar 26 2016, David Bremner <david@tethera.net> wrote:
>
>
> notmuch_bool_t
> notmuch_options_get (const char *name) {
> if (STRNCMP_LITERAL (name, "compact") == 0) {
> return HAVE_XAPIAN_COMPACT;
> // or return HAVE_XAPIAN_COMPACT? TRUE: FALSE;
> // or return !! HAVE_XAPIAN_COMPACT;
> } else if (STRNCMP_LITERAL (name, "field_processor") == 0) {
> return HAVE_XAPIAN_FIELD_PROCESSOR;
> } else {
> return FALSE;
> }
> }
I have no real objection to this. It's a bit slower than matching
against an enum, but I don't see this being in any performance critical
paths.
>
> ... note: if this notmuch_options_present () function prevails, fix
> the later #if check ;D
>
> ... and
>
> void
> _notmuch_config_list_options () {
> printf("options.compact=%s\n",
> HAVE_XAPIAN_COMPACT ? "true" : "false");
> printf("options.field_processor=%s\n",
> HAVE_XAPIAN_FIELD_PROCESSOR ? "true" : "false");
> }
>
This I'm not so sure about. There's a subtle distinction between what is
supported in the CLI and what is supported in the lib, and we really
want the latter. So I do think _some_ library routine is needed.
> (kept the 'options' naming and prefix in this context)
As far as the naming goes, I agree options is a bit weird. We could call
these things library features and the existing ones database features,
since the the latter are not really exposed to users. Other possible
words are
trait
attribute
property
^ permalink raw reply [flat|nested] 38+ messages in thread* Re: [Patch v2 04/13] lib/cli: add library API / CLI for compile time options
2016-04-30 11:53 ` David Bremner
@ 2016-05-01 17:48 ` Tomi Ollila
0 siblings, 0 replies; 38+ messages in thread
From: Tomi Ollila @ 2016-05-01 17:48 UTC (permalink / raw)
To: notmuch
On Sat, Apr 30 2016, David Bremner <david@tethera.net> wrote:
> Tomi Ollila <tomi.ollila@iki.fi> writes:
>
>> On Sat, Mar 26 2016, David Bremner <david@tethera.net> wrote:
>>
>>
>> ... note: if this notmuch_options_present () function prevails, fix
>> the later #if check ;D
>>
>> ... and
>>
>> void
>> _notmuch_config_list_options () {
>> printf("options.compact=%s\n",
>> HAVE_XAPIAN_COMPACT ? "true" : "false");
>> printf("options.field_processor=%s\n",
>> HAVE_XAPIAN_FIELD_PROCESSOR ? "true" : "false");
>> }
>>
>
> This I'm not so sure about. There's a subtle distinction between what is
> supported in the CLI and what is supported in the lib, and we really
> want the latter. So I do think _some_ library routine is needed.
Ah! this is the one thing I did not catch -- we need to know what _library_
supports from cli/bindings/whatnot. This makes most of the things I wrote
in my email irrelevant ;/
You seem to have sent new series so I stop commenting this and concentrate
on that...
Tomi
^ permalink raw reply [flat|nested] 38+ messages in thread
* [Patch v2 05/13] configure: check directly for xapian compaction API
2016-03-26 17:57 v2 of libconfig, date:foo, and named query patches David Bremner
` (3 preceding siblings ...)
2016-03-26 17:57 ` [Patch v2 04/13] lib/cli: add library API / CLI for compile time options David Bremner
@ 2016-03-26 17:57 ` David Bremner
2016-03-26 17:57 ` [Patch v2 06/13] lib: provide config API David Bremner
` (8 subsequent siblings)
13 siblings, 0 replies; 38+ messages in thread
From: David Bremner @ 2016-03-26 17:57 UTC (permalink / raw)
To: notmuch
This is consistent with the check for FieldProcessor, and probably a bit
more robust.
---
configure | 21 +++++++++++----------
1 file changed, 11 insertions(+), 10 deletions(-)
diff --git a/configure b/configure
index c48f7ce..03f28d5 100755
--- a/configure
+++ b/configure
@@ -359,17 +359,18 @@ fi
have_xapian_compact=0
have_xapian_field_processor=0
if [ ${have_xapian} = "1" ]; then
- # Compaction is only supported on Xapian > 1.2.6
printf "Checking for Xapian compaction support... "
- case "${xapian_version}" in
- 0.*|1.[01].*|1.2.[0-5])
- printf "No (only available with Xapian > 1.2.6).\n" ;;
- [1-9]*.[0-9]*.[0-9]*)
- have_xapian_compact=1
- printf "Yes.\n" ;;
- *)
- printf "Unknown version.\n" ;;
- esac
+ cat>_compact.cc<<EOF
+#include <xapian.h>
+class TestCompactor : public Xapian::Compactor { };
+EOF
+ if ${CXX} ${CXXLAGS} ${xapian_cxxflags} -c _compact.cc -o _compact.o > /dev/null 2>&1
+ then
+ have_xapian_compact=1
+ printf "Yes.\n"
+ else
+ printf "No.\n"
+ fi
printf "Checking for Xapian FieldProcessor API... "
cat>_field_processor.cc<<EOF
--
2.6.4
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Patch v2 06/13] lib: provide config API
2016-03-26 17:57 v2 of libconfig, date:foo, and named query patches David Bremner
` (4 preceding siblings ...)
2016-03-26 17:57 ` [Patch v2 05/13] configure: check directly for xapian compaction API David Bremner
@ 2016-03-26 17:57 ` David Bremner
2016-03-26 17:57 ` [Patch v2 07/13] lib: config list iterators David Bremner
` (7 subsequent siblings)
13 siblings, 0 replies; 38+ messages in thread
From: David Bremner @ 2016-03-26 17:57 UTC (permalink / raw)
To: notmuch
This is a thin wrapper around the Xapian metadata API. The job of this
layer is to keep the config key value pairs from colliding with other
metadata by transparently prefixing the keys, along with the usual glue
to provide a C interface.
The split of _get_config into two functions is to allow returning of the
return value with different memory ownership semantics.
---
lib/Makefile.local | 1 +
lib/config.cc | 90 ++++++++++++++++++++++++++++++++++++++++++++++++++
lib/notmuch.h | 20 +++++++++++
test/T590-libconfig.sh | 58 ++++++++++++++++++++++++++++++++
4 files changed, 169 insertions(+)
create mode 100644 lib/config.cc
create mode 100755 test/T590-libconfig.sh
diff --git a/lib/Makefile.local b/lib/Makefile.local
index 4ad0158..96899bf 100644
--- a/lib/Makefile.local
+++ b/lib/Makefile.local
@@ -49,6 +49,7 @@ libnotmuch_cxx_srcs = \
$(dir)/index.cc \
$(dir)/message.cc \
$(dir)/query.cc \
+ $(dir)/config.cc \
$(dir)/thread.cc
libnotmuch_modules := $(libnotmuch_c_srcs:.c=.o) $(libnotmuch_cxx_srcs:.cc=.o)
diff --git a/lib/config.cc b/lib/config.cc
new file mode 100644
index 0000000..af00d6f
--- /dev/null
+++ b/lib/config.cc
@@ -0,0 +1,90 @@
+/* metadata.cc - API for database metadata
+ *
+ * Copyright © 2015 David Bremner
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see http://www.gnu.org/licenses/ .
+ *
+ * Author: David Bremner <david@tethera.net>
+ */
+
+#include "notmuch.h"
+#include "notmuch-private.h"
+#include "database-private.h"
+
+static const std::string CONFIG_PREFIX="C";
+
+notmuch_status_t
+notmuch_database_set_config (notmuch_database_t *notmuch,
+ const char *key,
+ const char *value)
+{
+ notmuch_status_t status;
+ Xapian::WritableDatabase *db;
+
+ status = _notmuch_database_ensure_writable (notmuch);
+ if (status)
+ return status;
+
+ try {
+ db = static_cast <Xapian::WritableDatabase *> (notmuch->xapian_db);
+ db->set_metadata (CONFIG_PREFIX+key, value);
+ } catch (const Xapian::Error &error) {
+ status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
+ notmuch->exception_reported = TRUE;
+ if (! notmuch->exception_reported) {
+ _notmuch_database_log (notmuch, "Error: A Xapian exception occurred setting metadata: %s\n",
+ error.get_msg().c_str());
+ }
+ }
+ return NOTMUCH_STATUS_SUCCESS;
+}
+
+static notmuch_status_t
+_metadata_value (notmuch_database_t *notmuch,
+ const char *key,
+ std::string &value)
+{
+ notmuch_status_t status = NOTMUCH_STATUS_SUCCESS;
+
+ try {
+ value = notmuch->xapian_db->get_metadata (CONFIG_PREFIX+key);
+ } catch (const Xapian::Error &error) {
+ status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
+ notmuch->exception_reported = TRUE;
+ if (! notmuch->exception_reported) {
+ _notmuch_database_log (notmuch, "Error: A Xapian exception occurred getting metadata: %s\n",
+ error.get_msg().c_str());
+ }
+ }
+ return status;
+}
+
+notmuch_status_t
+notmuch_database_get_config (notmuch_database_t *notmuch,
+ const char *key,
+ char **value) {
+ std::string strval;
+ notmuch_status_t status;
+
+ if (!value)
+ return NOTMUCH_STATUS_NULL_POINTER;
+
+ status = _metadata_value (notmuch, key, strval);
+ if (status)
+ return status;
+
+ *value = strdup (strval.c_str ());
+
+ return NOTMUCH_STATUS_SUCCESS;
+}
diff --git a/lib/notmuch.h b/lib/notmuch.h
index b29dd5f..e416fb4 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -1838,6 +1838,26 @@ notmuch_filenames_move_to_next (notmuch_filenames_t *filenames);
void
notmuch_filenames_destroy (notmuch_filenames_t *filenames);
+
+/**
+ * set config 'key' to 'value'
+ *
+ */
+notmuch_status_t
+notmuch_database_set_config (notmuch_database_t *db, const char *key, const char *value);
+
+/**
+ * retrieve config item 'key', assign to 'value'
+ *
+ * keys which have not been previously set with n_d_set_config will
+ * return an empty string.
+ *
+ * return value is allocated by malloc and should be freed by the
+ * caller.
+ */
+notmuch_status_t
+notmuch_database_get_config (notmuch_database_t *db, const char *key, char **value);
+
typedef enum {
NOTMUCH_OPTION_COMPACT = 1,
NOTMUCH_OPTION_FIELD_PROCESSOR = 2
diff --git a/test/T590-libconfig.sh b/test/T590-libconfig.sh
new file mode 100755
index 0000000..85e4497
--- /dev/null
+++ b/test/T590-libconfig.sh
@@ -0,0 +1,58 @@
+#!/usr/bin/env bash
+test_description="library config API"
+
+. ./test-lib.sh || exit 1
+
+add_email_corpus
+
+cat <<EOF > c_head
+#include <stdio.h>
+#include <string.h>
+#include <stdlib.h>
+#include <notmuch.h>
+
+void run(int line, notmuch_status_t ret)
+{
+ if (ret) {
+ fprintf (stderr, "line %d: %s\n", line, ret);
+ exit (1);
+ }
+}
+
+#define RUN(v) run(__LINE__, v);
+
+int main (int argc, char** argv)
+{
+ notmuch_database_t *db;
+ char *val;
+ notmuch_status_t stat;
+
+ RUN(notmuch_database_open (argv[1], NOTMUCH_DATABASE_MODE_READ_WRITE, &db));
+
+EOF
+
+cat <<EOF > c_tail
+ RUN(notmuch_database_destroy(db));
+}
+EOF
+
+test_begin_subtest "notmuch_database_{set,get}_config"
+cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR}
+{
+ RUN(notmuch_database_set_config (db, "testkey1", "testvalue1"));
+ RUN(notmuch_database_set_config (db, "testkey2", "testvalue2"));
+ RUN(notmuch_database_get_config (db, "testkey1", &val));
+ printf("testkey1 = %s\n", val);
+ RUN(notmuch_database_get_config (db, "testkey2", &val));
+ printf("testkey2 = %s\n", val);
+}
+EOF
+cat <<'EOF' >EXPECTED
+== stdout ==
+testkey1 = testvalue1
+testkey2 = testvalue2
+== stderr ==
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_done
--
2.6.4
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Patch v2 07/13] lib: config list iterators
2016-03-26 17:57 v2 of libconfig, date:foo, and named query patches David Bremner
` (5 preceding siblings ...)
2016-03-26 17:57 ` [Patch v2 06/13] lib: provide config API David Bremner
@ 2016-03-26 17:57 ` David Bremner
2016-03-26 17:57 ` [Patch v2 08/13] CLI: add print_status_database David Bremner
` (6 subsequent siblings)
13 siblings, 0 replies; 38+ messages in thread
From: David Bremner @ 2016-03-26 17:57 UTC (permalink / raw)
To: notmuch
Since xapian provides the ability to restrict the iterator to a given
prefix, we expose this ability to the user. Otherwise we mimic the other
iterator interfances in notmuch (e.g. tags.c).
---
lib/config.cc | 104 +++++++++++++++++++++++++++++++++++++++++++++++++
lib/notmuch.h | 44 +++++++++++++++++++++
test/T590-libconfig.sh | 60 ++++++++++++++++++++++++++++
3 files changed, 208 insertions(+)
diff --git a/lib/config.cc b/lib/config.cc
index af00d6f..e581f32 100644
--- a/lib/config.cc
+++ b/lib/config.cc
@@ -24,6 +24,19 @@
static const std::string CONFIG_PREFIX="C";
+struct _notmuch_config_list {
+ notmuch_database_t *notmuch;
+ Xapian::TermIterator *iterator;
+ char *current_key;
+ char *current_val;
+};
+
+static int
+_notmuch_config_list_destroy (notmuch_config_list_t *list) {
+ delete list->iterator;
+ return 0;
+}
+
notmuch_status_t
notmuch_database_set_config (notmuch_database_t *notmuch,
const char *key,
@@ -88,3 +101,94 @@ notmuch_database_get_config (notmuch_database_t *notmuch,
return NOTMUCH_STATUS_SUCCESS;
}
+
+notmuch_status_t
+notmuch_database_get_config_list (notmuch_database_t *notmuch,
+ const char *prefix,
+ notmuch_config_list_t **out)
+{
+ notmuch_config_list_t *list = NULL;
+ notmuch_status_t status = NOTMUCH_STATUS_SUCCESS;
+
+ list = talloc (notmuch, notmuch_config_list_t);
+ if (!list) {
+ status = NOTMUCH_STATUS_OUT_OF_MEMORY;
+ goto DONE;
+ }
+
+ talloc_set_destructor (list, _notmuch_config_list_destroy);
+ list->iterator = new Xapian::TermIterator;
+ list->notmuch = notmuch;
+ list->current_key = NULL;
+ list->current_val = NULL;
+
+ try {
+
+ *list->iterator = notmuch->xapian_db->metadata_keys_begin (CONFIG_PREFIX + (prefix ? prefix : ""));
+
+ } catch (const Xapian::Error &error) {
+ _notmuch_database_log (notmuch, "A Xapian exception occurred getting metadata iterator: %s.\n",
+ error.get_msg().c_str());
+ notmuch->exception_reported = TRUE;
+ status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
+ }
+
+ *out = list;
+
+ DONE:
+ if (status && list)
+ talloc_free (list);
+
+ return status;
+}
+
+notmuch_bool_t
+notmuch_config_list_valid (notmuch_config_list_t *metadata)
+{
+ if (*(metadata->iterator) == metadata->notmuch->xapian_db->metadata_keys_end())
+ return FALSE;
+
+ return TRUE;
+}
+
+const char *
+notmuch_config_list_key (notmuch_config_list_t *list)
+{
+ if (list->current_key)
+ talloc_free (list->current_key);
+
+ list->current_key = talloc_strdup (list, (**(list->iterator)).c_str () + CONFIG_PREFIX.length ());
+
+ return list->current_key;
+}
+
+const char *
+notmuch_config_list_value (notmuch_config_list_t *list)
+{
+ std::string strval;
+ notmuch_status_t status;
+ const char *key = notmuch_config_list_key (list);
+
+ /* TODO: better error reporting?? */
+ status = _metadata_value (list->notmuch, key, strval);
+ if (status)
+ return NULL;
+
+ if (list->current_val)
+ talloc_free(list->current_val);
+
+ list->current_val = talloc_strdup(list, strval.c_str ());
+ return list->current_val;
+}
+
+void
+notmuch_config_list_move_to_next (notmuch_config_list_t *list)
+{
+ (*(list->iterator))++;
+}
+
+void
+notmuch_config_list_destroy (notmuch_config_list_t *list)
+{
+ talloc_free (list);
+}
diff --git a/lib/notmuch.h b/lib/notmuch.h
index e416fb4..2278822 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -206,6 +206,7 @@ typedef struct _notmuch_message notmuch_message_t;
typedef struct _notmuch_tags notmuch_tags_t;
typedef struct _notmuch_directory notmuch_directory_t;
typedef struct _notmuch_filenames notmuch_filenames_t;
+typedef struct _notmuch_config_list notmuch_config_list_t;
#endif /* __DOXYGEN__ */
/**
@@ -1858,6 +1859,49 @@ notmuch_database_set_config (notmuch_database_t *db, const char *key, const char
notmuch_status_t
notmuch_database_get_config (notmuch_database_t *db, const char *key, char **value);
+/**
+ * Create an iterator for all config items with keys matching a given prefix
+ */
+notmuch_status_t
+notmuch_database_get_config_list (notmuch_database_t *db, const char *prefix, notmuch_config_list_t **out);
+
+/**
+ * Is 'config_list' iterator valid (i.e. _key, _value, _move_to_next can be called).
+ */
+notmuch_bool_t
+notmuch_config_list_valid (notmuch_config_list_t *config_list);
+
+/**
+ * return key for current config pair
+ *
+ * return value is owned by the iterator, and will be destroyed by the
+ * next call to notmuch_config_list_key or notmuch_config_list_destroy.
+ */
+const char *
+notmuch_config_list_key (notmuch_config_list_t *config_list);
+
+/**
+ * return 'value' for current config pair
+ *
+ * return value is owned by the iterator, and will be destroyed by the
+ * next call to notmuch_config_list_value or notmuch config_list_destroy
+ */
+const char *
+notmuch_config_list_value (notmuch_config_list_t *config_list);
+
+
+/**
+ * move 'config_list' iterator to the next pair
+ */
+void
+notmuch_config_list_move_to_next (notmuch_config_list_t *config_list);
+
+/**
+ * free any resources held by 'config_list'
+ */
+void
+notmuch_config_list_destroy (notmuch_config_list_t *config_list);
+
typedef enum {
NOTMUCH_OPTION_COMPACT = 1,
NOTMUCH_OPTION_FIELD_PROCESSOR = 2
diff --git a/test/T590-libconfig.sh b/test/T590-libconfig.sh
index 85e4497..8ca6883 100755
--- a/test/T590-libconfig.sh
+++ b/test/T590-libconfig.sh
@@ -55,4 +55,64 @@ testkey2 = testvalue2
EOF
test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "notmuch_database_get_config_list: empty list"
+cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR}
+{
+ notmuch_config_list_t *list;
+ RUN(notmuch_database_get_config_list (db, "nonexistent", &list));
+ printf("valid = %d\n", notmuch_config_list_valid (list));
+ notmuch_config_list_destroy (list);
+}
+EOF
+cat <<'EOF' >EXPECTED
+== stdout ==
+valid = 0
+== stderr ==
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+
+test_begin_subtest "notmuch_database_get_config_list: all pairs"
+cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR}
+{
+ notmuch_config_list_t *list;
+ RUN(notmuch_database_set_config (db, "zzzafter", "afterval"));
+ RUN(notmuch_database_set_config (db, "aaabefore", "beforeval"));
+ RUN(notmuch_database_get_config_list (db, "", &list));
+ for (; notmuch_config_list_valid (list); notmuch_config_list_move_to_next (list)) {
+ printf("%s %s\n", notmuch_config_list_key (list), notmuch_config_list_value(list));
+ }
+ notmuch_config_list_destroy (list);
+}
+EOF
+cat <<'EOF' >EXPECTED
+== stdout ==
+aaabefore beforeval
+testkey1 testvalue1
+testkey2 testvalue2
+zzzafter afterval
+== stderr ==
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "notmuch_database_get_config_list: one prefix"
+cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR}
+{
+ notmuch_config_list_t *list;
+ RUN(notmuch_database_get_config_list (db, "testkey", &list));
+ for (; notmuch_config_list_valid (list); notmuch_config_list_move_to_next (list)) {
+ printf("%s %s\n", notmuch_config_list_key (list), notmuch_config_list_value(list));
+ }
+ notmuch_config_list_destroy (list);
+}
+EOF
+cat <<'EOF' >EXPECTED
+== stdout ==
+testkey1 testvalue1
+testkey2 testvalue2
+== stderr ==
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
test_done
--
2.6.4
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Patch v2 08/13] CLI: add print_status_database
2016-03-26 17:57 v2 of libconfig, date:foo, and named query patches David Bremner
` (6 preceding siblings ...)
2016-03-26 17:57 ` [Patch v2 07/13] lib: config list iterators David Bremner
@ 2016-03-26 17:57 ` David Bremner
2016-03-27 20:25 ` [PATCH] nmbug: ignore # comments David Bremner
2016-03-26 17:57 ` [Patch v2 09/13] CLI: add optional config data to dump output David Bremner
` (5 subsequent siblings)
13 siblings, 1 reply; 38+ messages in thread
From: David Bremner @ 2016-03-26 17:57 UTC (permalink / raw)
To: notmuch
This could probably be used at quite a few places in the existing code,
but in the immediate future I plan to use in some new code in
notmuch-dump
---
notmuch-client.h | 5 +++++
status.c | 17 +++++++++++++++++
2 files changed, 22 insertions(+)
diff --git a/notmuch-client.h b/notmuch-client.h
index 18e6c60..b3d0b66 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -462,6 +462,11 @@ print_status_query (const char *loc,
const notmuch_query_t *query,
notmuch_status_t status);
+notmuch_status_t
+print_status_database (const char *loc,
+ const notmuch_database_t *database,
+ notmuch_status_t status);
+
#include "command-line-arguments.h"
extern char *notmuch_requested_db_uuid;
diff --git a/status.c b/status.c
index 8fa81cb..45d3fb4 100644
--- a/status.c
+++ b/status.c
@@ -19,3 +19,20 @@ print_status_query (const char *loc,
}
return status;
}
+
+notmuch_status_t
+print_status_database (const char *loc,
+ const notmuch_database_t *notmuch,
+ notmuch_status_t status)
+{
+ if (status) {
+ const char *msg;
+
+ fprintf (stderr, "%s: %s\n", loc,
+ notmuch_status_to_string (status));
+ msg = notmuch_database_status_string (notmuch);
+ if (msg)
+ fputs (msg, stderr);
+ }
+ return status;
+}
--
2.6.4
^ permalink raw reply related [flat|nested] 38+ messages in thread* [PATCH] nmbug: ignore # comments
2016-03-26 17:57 ` [Patch v2 08/13] CLI: add print_status_database David Bremner
@ 2016-03-27 20:25 ` David Bremner
2016-03-27 20:38 ` W. Trevor King
2016-03-28 12:33 ` David Bremner
0 siblings, 2 replies; 38+ messages in thread
From: David Bremner @ 2016-03-27 20:25 UTC (permalink / raw)
To: David Bremner, notmuch
Lines starting with # have always (for a long time, anyway) been ignored
by notmuch-restore, but have not been generated by notmuch-dump
previously. In order to make nmbug robust against such output, ignore
comment lines.
---
Even if we decide to change the default output of notmuch dump to not
contain any comments, this fix will be reasonable in terms of
increasing robustness against future breakage.
devel/nmbug/nmbug | 2 ++
1 file changed, 2 insertions(+)
diff --git a/devel/nmbug/nmbug b/devel/nmbug/nmbug
index 81f582c..0787b2b 100755
--- a/devel/nmbug/nmbug
+++ b/devel/nmbug/nmbug
@@ -608,6 +608,8 @@ def _index_tags():
stdin=_subprocess.PIPE,
additional_env={'GIT_INDEX_FILE': path}) as git:
for line in notmuch.stdout:
+ if line.strip().startswith('#'):
+ continue
(tags_string, id) = [_.strip() for _ in line.split(' -- id:')]
tags = [
_unquote(tag[len(prefix):])
--
2.6.4
^ permalink raw reply related [flat|nested] 38+ messages in thread* Re: [PATCH] nmbug: ignore # comments
2016-03-27 20:25 ` [PATCH] nmbug: ignore # comments David Bremner
@ 2016-03-27 20:38 ` W. Trevor King
2016-03-28 7:14 ` Tomi Ollila
2016-03-28 12:33 ` David Bremner
1 sibling, 1 reply; 38+ messages in thread
From: W. Trevor King @ 2016-03-27 20:38 UTC (permalink / raw)
To: David Bremner; +Cc: notmuch
[-- Attachment #1: Type: text/plain, Size: 391 bytes --]
On Sun, Mar 27, 2016 at 05:25:11PM -0300, David Bremner wrote:
> for line in notmuch.stdout:
> + if line.strip().startswith('#'):
> + continue
Looks good to me.
Cheers,
Trevor
--
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 38+ messages in thread* Re: [PATCH] nmbug: ignore # comments
2016-03-27 20:25 ` [PATCH] nmbug: ignore # comments David Bremner
2016-03-27 20:38 ` W. Trevor King
@ 2016-03-28 12:33 ` David Bremner
1 sibling, 0 replies; 38+ messages in thread
From: David Bremner @ 2016-03-28 12:33 UTC (permalink / raw)
To: notmuch
David Bremner <david@tethera.net> writes:
> Lines starting with # have always (for a long time, anyway) been ignored
> by notmuch-restore, but have not been generated by notmuch-dump
> previously. In order to make nmbug robust against such output, ignore
> comment lines.
> ---
pushed.
d
^ permalink raw reply [flat|nested] 38+ messages in thread
* [Patch v2 09/13] CLI: add optional config data to dump output.
2016-03-26 17:57 v2 of libconfig, date:foo, and named query patches David Bremner
` (7 preceding siblings ...)
2016-03-26 17:57 ` [Patch v2 08/13] CLI: add print_status_database David Bremner
@ 2016-03-26 17:57 ` David Bremner
2016-03-26 18:13 ` David Bremner
2016-03-26 17:57 ` [Patch v2 10/13] CLI: optionally restore config data David Bremner
` (4 subsequent siblings)
13 siblings, 1 reply; 38+ messages in thread
From: David Bremner @ 2016-03-26 17:57 UTC (permalink / raw)
To: notmuch
Note that it changes the default dump output format, but doesn't break
existing notmuch-restore. It might break user scripts though.
---
doc/man1/notmuch-dump.rst | 17 ++++++++++
notmuch-client.h | 8 +++++
notmuch-dump.c | 80 +++++++++++++++++++++++++++++++++++++++++++++--
notmuch-new.c | 2 +-
test/T150-tagging.sh | 8 ++---
test/T240-dump-restore.sh | 14 ++++-----
test/T590-libconfig.sh | 17 ++++++++++
test/test-lib.sh | 6 ++++
8 files changed, 137 insertions(+), 15 deletions(-)
diff --git a/doc/man1/notmuch-dump.rst b/doc/man1/notmuch-dump.rst
index a37c337..eda9e07 100644
--- a/doc/man1/notmuch-dump.rst
+++ b/doc/man1/notmuch-dump.rst
@@ -71,6 +71,23 @@ Supported options for **dump** include
characters. Note also that tags with spaces will not be
correctly restored with this format.
+ ``--include=(config|tags)``
+
+ Control what kind of metadata is included in the output.
+
+ **config**
+
+ Output configuration data stored in the database. Each line
+ starts with "#@ ", followed by a space seperated key-value
+ pair. Both key and value are hex encoded if needed.
+
+ **tags**
+
+ Output per-message metadata, namely tags. See *format* above
+ for description of the output.
+
+ The default is to include both tags and configuration information
+
``--output=``\ <filename>
Write output to given file instead of stdout.
diff --git a/notmuch-client.h b/notmuch-client.h
index b3d0b66..ae6f124 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -446,11 +446,19 @@ typedef enum dump_formats {
DUMP_FORMAT_SUP
} dump_format_t;
+typedef enum dump_includes {
+ DUMP_INCLUDE_TAGS=1,
+ DUMP_INCLUDE_CONFIG=2,
+} dump_include_t;
+
+#define NOTMUCH_DUMP_VERSION 2
+
int
notmuch_database_dump (notmuch_database_t *notmuch,
const char *output_file_name,
const char *query_str,
dump_format_t output_format,
+ dump_include_t include,
notmuch_bool_t gzip_output);
/* If status is non-zero (i.e. error) print appropriate
diff --git a/notmuch-dump.c b/notmuch-dump.c
index 829781f..a6cf810 100644
--- a/notmuch-dump.c
+++ b/notmuch-dump.c
@@ -23,16 +23,82 @@
#include "string-util.h"
#include <zlib.h>
+static int
+database_dump_config (notmuch_database_t *notmuch, gzFile output)
+{
+ notmuch_config_list_t *list;
+ int ret = EXIT_FAILURE;
+ char *buffer = NULL;
+ size_t buffer_size = 0;
+
+ if (print_status_database ("notmuch dump", notmuch,
+ notmuch_database_get_config_list (notmuch, NULL, &list)))
+ goto DONE;
+
+ for (; notmuch_config_list_valid (list); notmuch_config_list_move_to_next (list)) {
+ if (hex_encode (notmuch, notmuch_config_list_key (list),
+ &buffer, &buffer_size) != HEX_SUCCESS) {
+ fprintf (stderr, "Error: failed to hex-encode config key %s\n",
+ notmuch_config_list_key (list));
+ goto DONE;
+ }
+ gzprintf (output, "#@ %s", buffer);
+
+ if (hex_encode (notmuch, notmuch_config_list_value (list),
+ &buffer, &buffer_size) != HEX_SUCCESS) {
+ fprintf (stderr, "Error: failed to hex-encode config value %s\n",
+ notmuch_config_list_value (list) );
+ goto DONE;
+ }
+
+ gzprintf (output, " %s\n", buffer);
+ }
+
+ ret = EXIT_SUCCESS;
+
+ DONE:
+ if (list)
+ notmuch_config_list_destroy (list);
+
+ if (buffer)
+ talloc_free (buffer);
+
+ return ret;
+}
+
+static void
+print_dump_header (gzFile output, int output_format, int include)
+{
+ gzprintf (output, "#notmuch-dump %s:%d %s%s%s\n",
+ (output_format == DUMP_FORMAT_SUP) ? "sup" : "batch-tag",
+ NOTMUCH_DUMP_VERSION,
+ (include & DUMP_INCLUDE_CONFIG) ? "config" : "",
+ (include & DUMP_INCLUDE_TAGS) && (include & DUMP_INCLUDE_CONFIG) ? "," : "",
+ (include & DUMP_INCLUDE_TAGS) ? "tags" : "");
+
+
+}
static int
database_dump_file (notmuch_database_t *notmuch, gzFile output,
- const char *query_str, int output_format)
+ const char *query_str, int output_format, int include)
{
notmuch_query_t *query;
notmuch_messages_t *messages;
notmuch_message_t *message;
notmuch_tags_t *tags;
+ print_dump_header (output, output_format, include);
+
+ if (include & DUMP_INCLUDE_CONFIG) {
+ if (print_status_database ("notmuch dump", notmuch,
+ database_dump_config(notmuch,output)))
+ return EXIT_FAILURE;
+ }
+
+ if (! (include & DUMP_INCLUDE_TAGS))
+ return EXIT_SUCCESS;
+
if (! query_str)
query_str = "";
@@ -130,6 +196,7 @@ notmuch_database_dump (notmuch_database_t *notmuch,
const char *output_file_name,
const char *query_str,
dump_format_t output_format,
+ dump_include_t include,
notmuch_bool_t gzip_output)
{
gzFile output = NULL;
@@ -164,7 +231,7 @@ notmuch_database_dump (notmuch_database_t *notmuch,
goto DONE;
}
- ret = database_dump_file (notmuch, output, query_str, output_format);
+ ret = database_dump_file (notmuch, output, query_str, output_format, include);
if (ret) goto DONE;
ret = gzflush (output, Z_FINISH);
@@ -226,6 +293,7 @@ notmuch_dump_command (notmuch_config_t *config, int argc, char *argv[])
int opt_index;
int output_format = DUMP_FORMAT_BATCH_TAG;
+ int include = 0;
notmuch_bool_t gzip_output = 0;
notmuch_opt_desc_t options[] = {
@@ -233,6 +301,9 @@ notmuch_dump_command (notmuch_config_t *config, int argc, char *argv[])
(notmuch_keyword_t []){ { "sup", DUMP_FORMAT_SUP },
{ "batch-tag", DUMP_FORMAT_BATCH_TAG },
{ 0, 0 } } },
+ { NOTMUCH_OPT_KEYWORD_FLAGS, &include, "include", 'I',
+ (notmuch_keyword_t []){ { "config", DUMP_INCLUDE_CONFIG },
+ { "tags", DUMP_INCLUDE_TAGS} } },
{ NOTMUCH_OPT_STRING, &output_file_name, "output", 'o', 0 },
{ NOTMUCH_OPT_BOOLEAN, &gzip_output, "gzip", 'z', 0 },
{ NOTMUCH_OPT_INHERIT, (void *) ¬much_shared_options, NULL, 0, 0 },
@@ -245,6 +316,9 @@ notmuch_dump_command (notmuch_config_t *config, int argc, char *argv[])
notmuch_process_shared_options (argv[0]);
+ if (include == 0)
+ include = DUMP_INCLUDE_CONFIG | DUMP_INCLUDE_TAGS;
+
if (opt_index < argc) {
query_str = query_string_from_args (notmuch, argc - opt_index, argv + opt_index);
if (query_str == NULL) {
@@ -254,7 +328,7 @@ notmuch_dump_command (notmuch_config_t *config, int argc, char *argv[])
}
ret = notmuch_database_dump (notmuch, output_file_name, query_str,
- output_format, gzip_output);
+ output_format, include, gzip_output);
notmuch_database_destroy (notmuch);
diff --git a/notmuch-new.c b/notmuch-new.c
index e503776..fd2ff82 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -1041,7 +1041,7 @@ notmuch_new_command (notmuch_config_t *config, int argc, char *argv[])
}
if (notmuch_database_dump (notmuch, backup_name, "",
- DUMP_FORMAT_BATCH_TAG, TRUE)) {
+ DUMP_FORMAT_BATCH_TAG, DUMP_INCLUDE_CONFIG | DUMP_INCLUDE_TAGS, TRUE)) {
fprintf (stderr, "Backup failed. Aborting upgrade.");
return EXIT_FAILURE;
}
diff --git a/test/T150-tagging.sh b/test/T150-tagging.sh
index 8adcabc..f1a6df0 100755
--- a/test/T150-tagging.sh
+++ b/test/T150-tagging.sh
@@ -188,7 +188,7 @@ cat <<EOF > EXPECTED
+%22%27%22%27%22%22%27%27 +inbox +tag4 +tag5 +unread -- id:msg-002@notmuch-test-suite
EOF
-notmuch dump --format=batch-tag | sort > OUTPUT
+NOTMUCH_DUMP_TAGS > OUTPUT
notmuch restore --format=batch-tag < BACKUP
test_expect_equal_file EXPECTED OUTPUT
@@ -209,7 +209,7 @@ cat <<EOF > EXPECTED
+%21@%23%20%24%25%5e%26%2a%29-_=+%5b%7b%5c%20%7c%3b%3a%27%20%22,.%3c%60%7e +inbox +tag5 +unread -- id:msg-001@notmuch-test-suite
EOF
-notmuch dump --format=batch-tag | sort > OUTPUT
+NOTMUCH_DUMP_TAGS > OUTPUT
notmuch restore --format=batch-tag < BACKUP
test_expect_equal_file EXPECTED OUTPUT
@@ -235,7 +235,7 @@ cat <<EOF > EXPECTED
+%2a@%7d%cf%b5%f4%85%80%adO3%da%a7 +=%e0%ac%95%c8%b3+%ef%aa%95%c8%a64w%c7%9d%c9%a2%cf%b3%d6%82%24B%c4%a9%c5%a1UX%ee%99%b0%27E7%ca%a4%d0%8b%5d +A%e1%a0%bc%de%8b%d5%b2V%d9%9b%f3%b5%a2%a3M%d8%a1u@%f0%a0%ac%948%7e%f0%ab%86%af%27 +L%df%85%ef%a1%a5m@%d3%96%c2%ab%d4%9f%ca%b8%f3%b3%a2%bf%c7%b1_u%d7%b4%c7%b1 +P%c4%98%2f +R +inbox +tag5 +unread +%7e%d1%8b%25%ec%a0%ae%d1%a0M%3b%e3%b6%b7%e9%a4%87%3c%db%9a%cc%a8%e1%96%9d +%c4%bf7%c7%ab9H%c4%99k%ea%91%bd%c3%8ck%e2%b3%8dk%c5%952V%e4%99%b2%d9%b3%e4%8b%bda%5b%24%c7%9b +%da%88=f%cc%b9I%ce%af%7b%c9%97%e3%b9%8bH%cb%92X%d2%8c6 +%dc%9crh%d2%86B%e5%97%a2%22t%ed%99%82d -- id:msg-001@notmuch-test-suite
EOF
-notmuch dump --format=batch-tag | sort > OUTPUT
+NOTMUCH_DUMP_TAGS > OUTPUT
notmuch restore --format=batch-tag < BACKUP
test_expect_equal_file EXPECTED OUTPUT
@@ -260,7 +260,7 @@ cat <<EOF > EXPECTED
+foo%3a%3abar%25 +found%3a%3ait +inbox +tag5 +unread +winner -- id:msg-001@notmuch-test-suite
EOF
-notmuch dump --format=batch-tag | sort > OUTPUT
+NOTMUCH_DUMP_TAGS > OUTPUT
notmuch restore --format=batch-tag < BACKUP
test_expect_equal_file EXPECTED OUTPUT
diff --git a/test/T240-dump-restore.sh b/test/T240-dump-restore.sh
index e6976ff..758d355 100755
--- a/test/T240-dump-restore.sh
+++ b/test/T240-dump-restore.sh
@@ -97,7 +97,7 @@ test_expect_equal_file dump.expected dump.actual
# Note, we assume all messages from cworth have a message-id
# containing cworth.org
-grep 'cworth[.]org' dump.expected > dump-cworth.expected
+(head -1 dump.expected ; grep 'cworth[.]org' dump.expected) > dump-cworth.expected
test_begin_subtest "dump -- from:cworth"
notmuch dump -- from:cworth > dump-dash-cworth.actual
@@ -118,16 +118,16 @@ notmuch search --output=messages from:cworth | sed s/^id:// |\
test_expect_equal_file OUTPUT EXPECTED
test_begin_subtest "format=batch-tag, dump sanity check."
-notmuch dump --format=sup from:cworth | cut -f1 -d' ' | \
+NOTMUCH_DUMP_TAGS --format=sup from:cworth | cut -f1 -d' ' | \
sort > EXPECTED.$test_count
-notmuch dump --format=batch-tag from:cworth | sed 's/^.*-- id://' | \
+NOTMUCH_DUMP_TAGS --format=batch-tag from:cworth | sed 's/^.*-- id://' | \
sort > OUTPUT.$test_count
test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
test_begin_subtest "format=batch-tag, missing newline"
printf "+a_tag_without_newline -- id:20091117232137.GA7669@griffis1.net" > IN
notmuch restore --accumulate < IN
-notmuch dump id:20091117232137.GA7669@griffis1.net > OUT
+NOTMUCH_DUMP_TAGS id:20091117232137.GA7669@griffis1.net > OUT
cat <<EOF > EXPECTED
+a_tag_without_newline +inbox +unread -- id:20091117232137.GA7669@griffis1.net
EOF
@@ -155,7 +155,7 @@ cat <<EOF >EXPECTED.$test_count
+ -- id:20091117232137.GA7669@griffis1.net
EOF
notmuch restore --format=batch-tag < EXPECTED.$test_count
-notmuch dump --format=batch-tag id:20091117232137.GA7669@griffis1.net > OUTPUT.$test_count
+NOTMUCH_DUMP_TAGS --format=batch-tag id:20091117232137.GA7669@griffis1.net > OUTPUT.$test_count
test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
tag1='comic_swear=$&^%$^%\\//-+$^%$'
@@ -217,9 +217,9 @@ notmuch dump --format=batch-tag > OUTPUT.$test_count
test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
test_begin_subtest 'format=batch-tag, checking encoded output'
-notmuch dump --format=batch-tag -- from:cworth |\
+NOTMUCH_DUMP_TAGS --format=batch-tag -- from:cworth |\
awk "{ print \"+$enc1 +$enc2 +$enc3 -- \" \$5 }" > EXPECTED.$test_count
-notmuch dump --format=batch-tag -- from:cworth > OUTPUT.$test_count
+NOTMUCH_DUMP_TAGS --format=batch-tag -- from:cworth > OUTPUT.$test_count
test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
test_begin_subtest 'restoring sane tags'
diff --git a/test/T590-libconfig.sh b/test/T590-libconfig.sh
index 8ca6883..5ea5300 100755
--- a/test/T590-libconfig.sh
+++ b/test/T590-libconfig.sh
@@ -115,4 +115,21 @@ testkey2 testvalue2
EOF
test_expect_equal_file EXPECTED OUTPUT
+test_begin_subtest "dump config"
+cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR}
+{
+ RUN(notmuch_database_set_config (db, "key with spaces", "value, with, spaces!"));
+}
+EOF
+notmuch dump --include=config >OUTPUT
+cat <<'EOF' >EXPECTED
+#notmuch-dump batch-tag:2 config
+#@ aaabefore beforeval
+#@ key%20with%20spaces value,%20with,%20spaces%21
+#@ testkey1 testvalue1
+#@ testkey2 testvalue2
+#@ zzzafter afterval
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
test_done
diff --git a/test/test-lib.sh b/test/test-lib.sh
index 49e6f40..98abbe2 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -673,6 +673,12 @@ NOTMUCH_NEW ()
notmuch new "${@}" | grep -v -E -e '^Processed [0-9]*( total)? file|Found [0-9]* total file'
}
+NOTMUCH_DUMP_TAGS ()
+{
+ # this relies on the default format being batch-tag, otherwise some tests will break
+ notmuch dump --include=tags "${@}" | sed '/^#/d' | sort
+}
+
notmuch_search_sanitize ()
{
perl -pe 's/("?thread"?: ?)("?)................("?)/\1\2XXX\3/'
--
2.6.4
^ permalink raw reply related [flat|nested] 38+ messages in thread* Re: [Patch v2 09/13] CLI: add optional config data to dump output.
2016-03-26 17:57 ` [Patch v2 09/13] CLI: add optional config data to dump output David Bremner
@ 2016-03-26 18:13 ` David Bremner
0 siblings, 0 replies; 38+ messages in thread
From: David Bremner @ 2016-03-26 18:13 UTC (permalink / raw)
To: notmuch
David Bremner <david@tethera.net> writes:
> Note that it changes the default dump output format, but doesn't break
> existing notmuch-restore. It might break user scripts though.
blush. One of the things it breaks is nmbug. The quick and dirty fix
would be to have nmbug ignore any comment lines in the dump output.
There is an option to the new dump (--include=tags) which omits the
config information, but it still adds a single header line. I'd be open
to the idea of omitting the header line if there is no config
information in the file.
The other question is if the current default of including configuration
information is a good idea. For the most obvious use case of backing up the
user's data, it seems like the right thing to do.
d
^ permalink raw reply [flat|nested] 38+ messages in thread
* [Patch v2 10/13] CLI: optionally restore config data.
2016-03-26 17:57 v2 of libconfig, date:foo, and named query patches David Bremner
` (8 preceding siblings ...)
2016-03-26 17:57 ` [Patch v2 09/13] CLI: add optional config data to dump output David Bremner
@ 2016-03-26 17:57 ` David Bremner
2016-03-26 17:57 ` [Patch v2 11/13] CLI: add notmuch-config support for named queries David Bremner
` (3 subsequent siblings)
13 siblings, 0 replies; 38+ messages in thread
From: David Bremner @ 2016-03-26 17:57 UTC (permalink / raw)
To: notmuch
The default to restore config data seems safest, especially since
currently we have no config data to mess up.
---
doc/man1/notmuch-restore.rst | 18 +++++++++++++++
notmuch-restore.c | 53 ++++++++++++++++++++++++++++++++++++++++++++
test/T590-libconfig.sh | 11 +++++++++
3 files changed, 82 insertions(+)
diff --git a/doc/man1/notmuch-restore.rst b/doc/man1/notmuch-restore.rst
index 362e262..87fa22e 100644
--- a/doc/man1/notmuch-restore.rst
+++ b/doc/man1/notmuch-restore.rst
@@ -50,6 +50,24 @@ Supported options for **restore** include
format, this heuristic, based the fact that batch-tag format
contains no parentheses, should be accurate.
+ ``--include=(config|tags)``
+
+ Control what kind of metadata is restored.
+
+ **config**
+
+ Restore configuration data to the database. Each configuration line starts
+ with "#@ ", followed by a space seperated key-value pair.
+ Both key and value are hex encoded if needed.
+
+ **tags**
+
+ Output per-message metadata, namely tags. See *format* above
+ for more details.
+
+ The default is to restore both tags and configuration
+ information
+
``--input=``\ <filename>
Read input from given file instead of stdin.
diff --git a/notmuch-restore.c b/notmuch-restore.c
index 9abc64f..e06fbde 100644
--- a/notmuch-restore.c
+++ b/notmuch-restore.c
@@ -24,6 +24,38 @@
#include "string-util.h"
#include "zlib-extra.h"
+static int
+process_config_line(notmuch_database_t *notmuch, const char* line){
+ const char *key_p, *val_p;
+ char *key, *val;
+ size_t key_len,val_len;
+ const char *delim=" \t\n";
+ int ret = EXIT_FAILURE;
+
+ void *local = talloc_new(NULL);
+
+ key_p = strtok_len_c(line, delim, &key_len);
+ val_p = strtok_len_c(key_p+key_len, delim, &val_len);
+
+ key = talloc_strndup(local, key_p, key_len);
+ val = talloc_strndup(local, val_p, val_len);
+ if (hex_decode_inplace (key) != HEX_SUCCESS ||
+ hex_decode_inplace (val) != HEX_SUCCESS ) {
+ fprintf (stderr, "hex decoding failure on line %s\n", line);
+ goto DONE;
+ }
+
+ if (print_status_database ("notmuch restore", notmuch,
+ notmuch_database_set_config (notmuch, key, val)))
+ goto DONE;
+
+ ret = EXIT_SUCCESS;
+
+ DONE:
+ talloc_free (local);
+ return ret;
+}
+
static regex_t regex;
/* Non-zero return indicates an error in retrieving the message,
@@ -137,6 +169,7 @@ notmuch_restore_command (notmuch_config_t *config, int argc, char *argv[])
int ret = 0;
int opt_index;
+ int include=0;
int input_format = DUMP_FORMAT_AUTO;
if (notmuch_database_open (notmuch_config_get_database_path (config),
@@ -152,6 +185,10 @@ notmuch_restore_command (notmuch_config_t *config, int argc, char *argv[])
{ "batch-tag", DUMP_FORMAT_BATCH_TAG },
{ "sup", DUMP_FORMAT_SUP },
{ 0, 0 } } },
+ { NOTMUCH_OPT_KEYWORD_FLAGS, &include, "include", 'I',
+ (notmuch_keyword_t []){ { "config", DUMP_INCLUDE_CONFIG },
+ { "tags", DUMP_INCLUDE_TAGS} } },
+
{ NOTMUCH_OPT_STRING, &input_file_name, "input", 'i', 0 },
{ NOTMUCH_OPT_BOOLEAN, &accumulate, "accumulate", 'a', 0 },
{ NOTMUCH_OPT_INHERIT, (void *) ¬much_shared_options, NULL, 0, 0 },
@@ -167,6 +204,10 @@ notmuch_restore_command (notmuch_config_t *config, int argc, char *argv[])
notmuch_process_shared_options (argv[0]);
notmuch_exit_if_unmatched_db_uuid (notmuch);
+ if (include == 0) {
+ include = DUMP_INCLUDE_CONFIG | DUMP_INCLUDE_TAGS;
+ }
+
name_for_error = input_file_name ? input_file_name : "stdin";
if (! accumulate)
@@ -225,11 +266,23 @@ notmuch_restore_command (notmuch_config_t *config, int argc, char *argv[])
ret = EXIT_FAILURE;
goto DONE;
}
+
+ if ((include & DUMP_INCLUDE_CONFIG) && line_len >= 2 && line[0] == '#' && line[1] == '@') {
+ ret = process_config_line(notmuch, line+2);
+ if (ret)
+ goto DONE;
+ }
+
} while ((line_len == 0) ||
(line[0] == '#') ||
/* the cast is safe because we checked about for line_len < 0 */
(strspn (line, " \t\n") == (unsigned)line_len));
+ if (! (include & DUMP_INCLUDE_TAGS)) {
+ ret = EXIT_SUCCESS;
+ goto DONE;
+ }
+
char *p;
for (p = line; (input_format == DUMP_FORMAT_AUTO) && *p; p++) {
if (*p == '(')
diff --git a/test/T590-libconfig.sh b/test/T590-libconfig.sh
index 5ea5300..9c1e566 100755
--- a/test/T590-libconfig.sh
+++ b/test/T590-libconfig.sh
@@ -132,4 +132,15 @@ cat <<'EOF' >EXPECTED
EOF
test_expect_equal_file EXPECTED OUTPUT
+test_begin_subtest "restore config"
+notmuch dump --include=config >EXPECTED
+cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR}
+{
+ RUN(notmuch_database_set_config (db, "testkey1", "mutatedvalue"));
+}
+EOF
+notmuch restore --include=config <EXPECTED
+notmuch dump --include=config >OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
test_done
--
2.6.4
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Patch v2 11/13] CLI: add notmuch-config support for named queries
2016-03-26 17:57 v2 of libconfig, date:foo, and named query patches David Bremner
` (9 preceding siblings ...)
2016-03-26 17:57 ` [Patch v2 10/13] CLI: optionally restore config data David Bremner
@ 2016-03-26 17:57 ` David Bremner
2016-03-26 17:57 ` [Patch v2 12/13] lib: make a global constant for query parser flags David Bremner
` (2 subsequent siblings)
13 siblings, 0 replies; 38+ messages in thread
From: David Bremner @ 2016-03-26 17:57 UTC (permalink / raw)
To: notmuch
Most of the infrastructure here is general, only the validation/dispatch
is hardcoded to a particular prefix.
A notable change in behaviour is that notmuch-config now opens the
database e.g. on every call to list, which fails with an error message
if the database doesn't exit yet.
---
doc/man1/notmuch-config.rst | 6 ++++
notmuch-config.c | 88 ++++++++++++++++++++++++++++++++++++++++++++-
test/Makefile.local | 2 +-
test/T030-config.sh | 12 ++++---
test/T600-named-queries.sh | 53 +++++++++++++++++++++++++++
test/test-lib.sh | 5 +++
6 files changed, 159 insertions(+), 7 deletions(-)
create mode 100755 test/T600-named-queries.sh
diff --git a/doc/man1/notmuch-config.rst b/doc/man1/notmuch-config.rst
index 150d764..c3470a8 100644
--- a/doc/man1/notmuch-config.rst
+++ b/doc/man1/notmuch-config.rst
@@ -138,6 +138,12 @@ The available configuration items are described below.
"compact" (see **notmuch-compact(1)**)
and "field_processor" (see **notmuch-search-terms(7)**).
+ **query.<name>**
+
+ Expansion for named query called <name>. See
+ **notmuch-search-terms(7)** for more information about named
+ queries.
+
ENVIRONMENT
===========
diff --git a/notmuch-config.c b/notmuch-config.c
index cfc549d..121fec6 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -751,6 +751,28 @@ _item_split (char *item, char **group, char **key)
}
#define OPTION_PREFIX "options."
+#define QUERY_PREFIX "query."
+
+static int
+_print_db_config(notmuch_config_t *config, const char *name)
+{
+ notmuch_database_t *notmuch;
+ char *val;
+
+ if (notmuch_database_open (notmuch_config_get_database_path (config),
+ NOTMUCH_DATABASE_MODE_READ_ONLY, ¬much))
+ return EXIT_FAILURE;
+
+ /* XXX Handle UUID mismatch? */
+
+ if (print_status_database ("notmuch config", notmuch,
+ notmuch_database_get_config (notmuch, name, &val)))
+ return EXIT_FAILURE;
+
+ puts (val);
+
+ return EXIT_SUCCESS;
+}
static int
notmuch_config_command_get (notmuch_config_t *config, char *item)
@@ -778,6 +800,8 @@ notmuch_config_command_get (notmuch_config_t *config, char *item)
} else if (STRNCMP_LITERAL (item, OPTION_PREFIX) == 0) {
printf ("%s\n",
notmuch_options_get (item + strlen (OPTION_PREFIX)) ? "true" : "false");
+ } else if (STRNCMP_LITERAL (item, QUERY_PREFIX) == 0) {
+ return _print_db_config (config, item);
} else {
char **value;
size_t i, length;
@@ -805,6 +829,39 @@ notmuch_config_command_get (notmuch_config_t *config, char *item)
}
static int
+_set_db_config(notmuch_config_t *config, const char *key, int argc, char **argv)
+{
+ notmuch_database_t *notmuch;
+ const char *val = "";
+
+ if (argc > 1) {
+ /* XXX handle lists? */
+ fprintf (stderr, "notmuch config set: at most one value expected for %s\n", key);
+ return EXIT_FAILURE;
+ }
+
+ if (argc > 0) {
+ val = argv[0];
+ }
+
+ if (notmuch_database_open (notmuch_config_get_database_path (config),
+ NOTMUCH_DATABASE_MODE_READ_WRITE, ¬much))
+ return EXIT_FAILURE;
+
+ /* XXX Handle UUID mismatch? */
+
+ if (print_status_database ("notmuch config", notmuch,
+ notmuch_database_set_config (notmuch, key, val)))
+ return EXIT_FAILURE;
+
+ if (print_status_database ("notmuch config", notmuch,
+ notmuch_database_close (notmuch)))
+ return EXIT_FAILURE;
+
+ return EXIT_SUCCESS;
+}
+
+static int
notmuch_config_command_set (notmuch_config_t *config, char *item, int argc, char *argv[])
{
char *group, *key;
@@ -814,6 +871,10 @@ notmuch_config_command_set (notmuch_config_t *config, char *item, int argc, char
return 1;
}
+ if (STRNCMP_LITERAL (item, QUERY_PREFIX) == 0) {
+ return _set_db_config (config, item, argc, argv);
+ }
+
if (_item_split (item, &group, &key))
return 1;
@@ -850,6 +911,31 @@ _notmuch_config_list_options () {
}
static int
+_list_db_config (notmuch_config_t *config)
+{
+ notmuch_database_t *notmuch;
+ notmuch_config_list_t *list;
+
+ if (notmuch_database_open (notmuch_config_get_database_path (config),
+ NOTMUCH_DATABASE_MODE_READ_ONLY, ¬much))
+ return EXIT_FAILURE;
+
+ /* XXX Handle UUID mismatch? */
+
+
+ if (print_status_database ("notmuch config", notmuch,
+ notmuch_database_get_config_list (notmuch, "", &list)))
+ return EXIT_FAILURE;
+
+ for (; notmuch_config_list_valid (list); notmuch_config_list_move_to_next (list)) {
+ printf("%s=%s\n", notmuch_config_list_key (list), notmuch_config_list_value(list));
+ }
+ notmuch_config_list_destroy (list);
+
+ return EXIT_SUCCESS;
+}
+
+static int
notmuch_config_command_list (notmuch_config_t *config)
{
char **groups;
@@ -885,7 +971,7 @@ notmuch_config_command_list (notmuch_config_t *config)
g_strfreev (groups);
_notmuch_config_list_options ();
- return 0;
+ return _list_db_config (config);
}
int
diff --git a/test/Makefile.local b/test/Makefile.local
index 30d420e..8c55441 100644
--- a/test/Makefile.local
+++ b/test/Makefile.local
@@ -19,7 +19,7 @@ $(dir)/hex-xcode: $(dir)/hex-xcode.o command-line-arguments.o util/libutil.a
$(call quiet,CC) $^ -o $@ $(LDFLAGS) $(TALLOC_LDFLAGS)
random_corpus_deps = $(dir)/random-corpus.o $(dir)/database-test.o \
- notmuch-config.o command-line-arguments.o \
+ notmuch-config.o status.o command-line-arguments.o \
lib/libnotmuch.a util/libutil.a \
parse-time-string/libparse-time-string.a
diff --git a/test/T030-config.sh b/test/T030-config.sh
index c37ba21..39ee885 100755
--- a/test/T030-config.sh
+++ b/test/T030-config.sh
@@ -43,10 +43,10 @@ notmuch config set foo.nonexistent
test_expect_equal "$(notmuch config get foo.nonexistent)" ""
test_begin_subtest "List all items"
-notmuch config set database.path "/canonical/path"
-output=$(notmuch config list | notmuch_options_sanitize)
-test_expect_equal "$output" "\
-database.path=/canonical/path
+notmuch config list 2>&1 | notmuch_config_sanitize > OUTPUT
+cat <<EOF > EXPECTED
+Error opening database at MAIL_DIR/.notmuch: No such file or directory
+database.path=MAIL_DIR
user.name=Notmuch Test Suite
user.primary_email=test_suite@notmuchmail.org
user.other_email=test_suite_other@notmuchmail.org;test_suite@otherdomain.org
@@ -58,7 +58,9 @@ crypto.gpg_path=gpg
foo.string=this is another string value
foo.list=this;is another;list value;
options.compact=something
-options.field_processor=something"
+options.field_processor=something
+EOF
+test_expect_equal_file EXPECTED OUTPUT
test_begin_subtest "Top level --config=FILE option"
cp "${NOTMUCH_CONFIG}" alt-config
diff --git a/test/T600-named-queries.sh b/test/T600-named-queries.sh
new file mode 100755
index 0000000..0922620
--- /dev/null
+++ b/test/T600-named-queries.sh
@@ -0,0 +1,53 @@
+#!/usr/bin/env bash
+test_description='named queries'
+. ./test-lib.sh || exit 1
+
+QUERYSTR="date:2009-11-18..2009-11-18 and tag:unread"
+
+test_expect_code 1 "error adding named query before initializing DB" \
+ "notmuch config set query.test \"$QUERYSTR\""
+
+add_email_corpus
+
+test_expect_success "adding named query" \
+ "notmuch config set query.test \"$QUERYSTR\""
+
+QUERYSTR2="query:test and subject:Maildir"
+test_expect_success "adding nested named query" \
+ "notmuch config set query.test2 \"$QUERYSTR2\""
+
+test_begin_subtest "retrieve named query"
+output=$(notmuch config get query.test)
+test_expect_equal "$QUERYSTR" "$output"
+
+test_begin_subtest "List all queries"
+notmuch config list | grep ^query | notmuch_config_sanitize > OUTPUT
+cat <<EOF > EXPECTED
+query.test=date:2009-11-18..2009-11-18 and tag:unread
+query.test2=query:test and subject:Maildir
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "dump named queries"
+notmuch dump | grep '^#@' > OUTPUT
+cat<<EOF > QUERIES.BEFORE
+#@ query.test date%3a2009-11-18..2009-11-18%20and%20tag%3aunread
+#@ query.test2 query%3atest%20and%20subject%3aMaildir
+EOF
+test_expect_equal_file QUERIES.BEFORE OUTPUT
+
+test_begin_subtest "delete named queries"
+notmuch dump > BEFORE
+notmuch config set query.test
+notmuch dump | grep '^#@' > OUTPUT
+cat<<EOF > EXPECTED
+#@ query.test2 query%3atest%20and%20subject%3aMaildir
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "restore named queries"
+notmuch restore < BEFORE
+notmuch dump | grep '^#@' > OUTPUT
+test_expect_equal_file QUERIES.BEFORE OUTPUT
+
+test_done
diff --git a/test/test-lib.sh b/test/test-lib.sh
index 98abbe2..00df2bd 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -745,6 +745,11 @@ notmuch_options_sanitize ()
sed 's/^options[.]\(.*\)=.*$/options.\1=something/'
}
+notmuch_config_sanitize ()
+{
+ notmuch_dir_sanitize | notmuch_options_sanitize
+}
+
# End of notmuch helper functions
# Use test_set_prereq to tell that a particular prerequisite is available.
--
2.6.4
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Patch v2 12/13] lib: make a global constant for query parser flags
2016-03-26 17:57 v2 of libconfig, date:foo, and named query patches David Bremner
` (10 preceding siblings ...)
2016-03-26 17:57 ` [Patch v2 11/13] CLI: add notmuch-config support for named queries David Bremner
@ 2016-03-26 17:57 ` David Bremner
2016-03-26 17:57 ` [Patch v2 13/13] lib: add support for named queries David Bremner
2016-04-01 23:57 ` Breaking a really long thread David Mazieres
13 siblings, 0 replies; 38+ messages in thread
From: David Bremner @ 2016-03-26 17:57 UTC (permalink / raw)
To: notmuch
It's already kindof gross that this is hardcoded in two different
places. We will also need these later in field processors calling back
into the query parser.
---
lib/database-private.h | 4 ++++
lib/query.cc | 16 ++--------------
2 files changed, 6 insertions(+), 14 deletions(-)
diff --git a/lib/database-private.h b/lib/database-private.h
index e1962f4..b5c1f90 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -144,6 +144,10 @@ operator&=(_notmuch_features &a, _notmuch_features b)
return a;
}
+#define NOTMUCH_QUERY_PARSER_FLAGS (Xapian::QueryParser::FLAG_BOOLEAN | Xapian::QueryParser::FLAG_PHRASE | \
+ Xapian::QueryParser::FLAG_LOVEHATE | Xapian::QueryParser::FLAG_BOOLEAN_ANY_CASE | \
+ Xapian::QueryParser::FLAG_WILDCARD | Xapian::QueryParser::FLAG_PURE_NOT)
+
struct _notmuch_database {
notmuch_bool_t exception_reported;
diff --git a/lib/query.cc b/lib/query.cc
index e627bfc..972a389 100644
--- a/lib/query.cc
+++ b/lib/query.cc
@@ -212,12 +212,6 @@ notmuch_query_search_messages_st (notmuch_query_t *query,
Xapian::Query string_query, final_query, exclude_query;
Xapian::MSet mset;
Xapian::MSetIterator iterator;
- unsigned int flags = (Xapian::QueryParser::FLAG_BOOLEAN |
- Xapian::QueryParser::FLAG_PHRASE |
- Xapian::QueryParser::FLAG_LOVEHATE |
- Xapian::QueryParser::FLAG_BOOLEAN_ANY_CASE |
- Xapian::QueryParser::FLAG_WILDCARD |
- Xapian::QueryParser::FLAG_PURE_NOT);
if (strcmp (query_string, "") == 0 ||
strcmp (query_string, "*") == 0)
@@ -225,7 +219,7 @@ notmuch_query_search_messages_st (notmuch_query_t *query,
final_query = mail_query;
} else {
string_query = notmuch->query_parser->
- parse_query (query_string, flags);
+ parse_query (query_string, NOTMUCH_QUERY_PARSER_FLAGS);
final_query = Xapian::Query (Xapian::Query::OP_AND,
mail_query, string_query);
}
@@ -565,12 +559,6 @@ notmuch_query_count_messages_st (notmuch_query_t *query, unsigned *count_out)
"mail"));
Xapian::Query string_query, final_query, exclude_query;
Xapian::MSet mset;
- unsigned int flags = (Xapian::QueryParser::FLAG_BOOLEAN |
- Xapian::QueryParser::FLAG_PHRASE |
- Xapian::QueryParser::FLAG_LOVEHATE |
- Xapian::QueryParser::FLAG_BOOLEAN_ANY_CASE |
- Xapian::QueryParser::FLAG_WILDCARD |
- Xapian::QueryParser::FLAG_PURE_NOT);
if (strcmp (query_string, "") == 0 ||
strcmp (query_string, "*") == 0)
@@ -578,7 +566,7 @@ notmuch_query_count_messages_st (notmuch_query_t *query, unsigned *count_out)
final_query = mail_query;
} else {
string_query = notmuch->query_parser->
- parse_query (query_string, flags);
+ parse_query (query_string, NOTMUCH_QUERY_PARSER_FLAGS);
final_query = Xapian::Query (Xapian::Query::OP_AND,
mail_query, string_query);
}
--
2.6.4
^ permalink raw reply related [flat|nested] 38+ messages in thread* [Patch v2 13/13] lib: add support for named queries
2016-03-26 17:57 v2 of libconfig, date:foo, and named query patches David Bremner
` (11 preceding siblings ...)
2016-03-26 17:57 ` [Patch v2 12/13] lib: make a global constant for query parser flags David Bremner
@ 2016-03-26 17:57 ` David Bremner
2016-04-01 23:57 ` Breaking a really long thread David Mazieres
13 siblings, 0 replies; 38+ messages in thread
From: David Bremner @ 2016-03-26 17:57 UTC (permalink / raw)
To: notmuch
This relies on the optional presense of xapian field processors, and the
library config API.
---
doc/man7/notmuch-search-terms.rst | 8 ++++++++
lib/Makefile.local | 1 +
lib/database-private.h | 1 +
lib/database.cc | 3 +++
lib/query-fp.cc | 42 +++++++++++++++++++++++++++++++++++++++
lib/query-fp.h | 42 +++++++++++++++++++++++++++++++++++++++
test/T600-named-queries.sh | 17 ++++++++++++++++
7 files changed, 114 insertions(+)
create mode 100644 lib/query-fp.cc
create mode 100644 lib/query-fp.h
diff --git a/doc/man7/notmuch-search-terms.rst b/doc/man7/notmuch-search-terms.rst
index 29e8b03..7474f53 100644
--- a/doc/man7/notmuch-search-terms.rst
+++ b/doc/man7/notmuch-search-terms.rst
@@ -56,6 +56,8 @@ indicate user-supplied values):
- lastmod:<initial-revision>..<final-revision>
+- query:<name>
+
The **from:** prefix is used to match the name or address of the sender
of an email message.
@@ -132,6 +134,11 @@ were added/removed or filenames changed). This is usually used in
conjunction with the **--uuid** argument to **notmuch search**
to find messages that have changed since an earlier query.
+The **query:** prefix allows queries to refer to previously saved
+queries added with **notmuch-config(1)**. Named queries are only
+available if notmuch is built with **Xapian Field Processors** (see
+below).
+
Operators
---------
@@ -385,6 +392,7 @@ notmuch was built against a sufficiently recent version of Xapian by running
Currently the following features require field processor support:
- non-range date queries, e.g. "date:today"
+- named queries e.g. "query:my_special_query"
SEE ALSO
========
diff --git a/lib/Makefile.local b/lib/Makefile.local
index 96899bf..fab1242 100644
--- a/lib/Makefile.local
+++ b/lib/Makefile.local
@@ -49,6 +49,7 @@ libnotmuch_cxx_srcs = \
$(dir)/index.cc \
$(dir)/message.cc \
$(dir)/query.cc \
+ $(dir)/query-fp.cc \
$(dir)/config.cc \
$(dir)/thread.cc
diff --git a/lib/database-private.h b/lib/database-private.h
index b5c1f90..5ab4001 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -182,6 +182,7 @@ struct _notmuch_database {
Xapian::ValueRangeProcessor *date_range_processor;
#if HAVE_XAPIAN_FIELD_PROCESSOR
Xapian::FieldProcessor *date_field_processor;
+ Xapian::FieldProcessor *query_field_processor;
#endif
Xapian::ValueRangeProcessor *last_mod_range_processor;
};
diff --git a/lib/database.cc b/lib/database.cc
index 0e3c90c..8df5555 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -20,6 +20,7 @@
#include "database-private.h"
#include "parse-time-vrp.h"
+#include "query-fp.h"
#include "string-util.h"
#include <iostream>
@@ -1005,6 +1006,8 @@ notmuch_database_open_verbose (const char *path,
* with a .. to the range processor */
notmuch->date_field_processor = new DateFieldProcessor();
notmuch->query_parser->add_boolean_prefix("date", notmuch->date_field_processor);
+ notmuch->query_field_processor = new QueryFieldProcessor (*notmuch->query_parser, notmuch);
+ notmuch->query_parser->add_boolean_prefix("query", notmuch->query_field_processor);
#endif
notmuch->last_mod_range_processor = new Xapian::NumberValueRangeProcessor (NOTMUCH_VALUE_LAST_MOD, "lastmod:");
diff --git a/lib/query-fp.cc b/lib/query-fp.cc
new file mode 100644
index 0000000..ee5a306
--- /dev/null
+++ b/lib/query-fp.cc
@@ -0,0 +1,42 @@
+/* query-fp.cc - "query:" field processor glue glue
+ *
+ * This file is part of notmuch.
+ *
+ * Copyright © 2016 David Bremner
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see http://www.gnu.org/licenses/ .
+ *
+ * Author: David Bremner <david@tethera.net>
+ */
+
+#include "database-private.h"
+#include "query-fp.h"
+#include <iostream>
+
+#if HAVE_XAPIAN_FIELD_PROCESSOR
+
+Xapian::Query QueryFieldProcessor::operator()(const std::string & name) {
+ std::string key = "query." + name;
+ char *expansion;
+ notmuch_status_t status;
+
+ status = notmuch_database_get_config (notmuch, key.c_str(), &expansion);
+ if (status) {
+ throw Xapian::QueryParserError("error looking up key" + name);
+
+ }
+
+ return parser.parse_query (expansion, NOTMUCH_QUERY_PARSER_FLAGS);
+}
+#endif
diff --git a/lib/query-fp.h b/lib/query-fp.h
new file mode 100644
index 0000000..67f8705
--- /dev/null
+++ b/lib/query-fp.h
@@ -0,0 +1,42 @@
+/* query-fp.h - query field processor glue
+ *
+ * This file is part of notmuch.
+ *
+ * Copyright © 2016 David Bremner
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see http://www.gnu.org/licenses/ .
+ *
+ * Author: David Bremner <david@tethera.net>
+ */
+
+#ifndef NOTMUCH_QUERY_FP_H
+#define NOTMUCH_QUERY_FP_H
+
+#include <xapian.h>
+#include "notmuch.h"
+
+#if HAVE_XAPIAN_FIELD_PROCESSOR
+class QueryFieldProcessor : public Xapian::FieldProcessor {
+ protected:
+ Xapian::QueryParser &parser;
+ notmuch_database_t *notmuch;
+
+ public:
+ QueryFieldProcessor (Xapian::QueryParser &parser_, notmuch_database_t *notmuch_)
+ : parser(parser_), notmuch(notmuch_) { };
+
+ Xapian::Query operator()(const std::string & str);
+};
+#endif
+#endif /* NOTMUCH_QUERY_FP_H */
diff --git a/test/T600-named-queries.sh b/test/T600-named-queries.sh
index 0922620..f0ae24f 100755
--- a/test/T600-named-queries.sh
+++ b/test/T600-named-queries.sh
@@ -50,4 +50,21 @@ notmuch restore < BEFORE
notmuch dump | grep '^#@' > OUTPUT
test_expect_equal_file QUERIES.BEFORE OUTPUT
+if [ $NOTMUCH_HAVE_XAPIAN_FIELD_PROCESSOR -eq 1 ]; then
+ test_begin_subtest "search named query"
+ notmuch search query:test > OUTPUT
+ notmuch search $QUERYSTR > EXPECTED
+ test_expect_equal_file EXPECTED OUTPUT
+
+ test_begin_subtest "search named query with other terms"
+ notmuch search query:test and subject:Maildir > OUTPUT
+ notmuch search $QUERYSTR and subject:Maildir > EXPECTED
+ test_expect_equal_file EXPECTED OUTPUT
+
+ test_begin_subtest "search nested named query"
+ notmuch search query:test2 > OUTPUT
+ notmuch search $QUERYSTR2 > EXPECTED
+ test_expect_equal_file EXPECTED OUTPUT
+fi
+
test_done
--
2.6.4
^ permalink raw reply related [flat|nested] 38+ messages in thread* Breaking a really long thread
2016-03-26 17:57 v2 of libconfig, date:foo, and named query patches David Bremner
` (12 preceding siblings ...)
2016-03-26 17:57 ` [Patch v2 13/13] lib: add support for named queries David Bremner
@ 2016-04-01 23:57 ` David Mazieres
2016-04-02 12:41 ` David Bremner
13 siblings, 1 reply; 38+ messages in thread
From: David Mazieres @ 2016-04-01 23:57 UTC (permalink / raw)
To: notmuch
For some reason, Google voice always includes a header of the form
"References: <+15555555555.@txt.google.com>" (where the 5s are my actual
phone number) when I get a text. The result is that every SMS I've ever
received is in a single thread containing many thousands of messages.
And of course huge threads are annoyingly slow to open, particularly to
read a single 140-character message.
Is there any way to break an existing thread (so as to start over with a
smaller thread), or otherwise to tweak the threading rules so that a
particular References header gets ignored. It's annoyingly slow to open
a thread with 10,000 messages just to read one SMS. I'm almost tempted
to mangle the messages on delivery and remove the References header
before notmuch sees them, but it would be nice to have a cleaner
solution, as there are other situations in which one might want to
"reset" a really long thread.
Thanks,
David
^ permalink raw reply [flat|nested] 38+ messages in thread* Re: Breaking a really long thread
2016-04-01 23:57 ` Breaking a really long thread David Mazieres
@ 2016-04-02 12:41 ` David Bremner
2016-04-02 13:56 ` David Mazieres
0 siblings, 1 reply; 38+ messages in thread
From: David Bremner @ 2016-04-02 12:41 UTC (permalink / raw)
To: David Mazieres expires 2016-06-30 PDT, notmuch
David Mazieres <dm-list-email-notmuch@scs.stanford.edu> writes:
> Is there any way to break an existing thread (so as to start over with a
> smaller thread), or otherwise to tweak the threading rules so that a
> particular References header gets ignored.
Currently there is no way to do this, as threads are "stateless"
i.e. created on the fly by _notmuch_create_thread based only on
immutable mail data. We'd have to be willing to dump/restore any
threading hints we added to the database, but we're already headed down
that path with the libconfig stuff anyway. Something we _almost_ have
the infrastructure in place for would be to blacklist a particular
reference (e.g. by regex) by storing the blacklist as database
metadata. That would solve your particular problem, although I'm not
sure it it's general enough. Keeping global state for all thread
breaking sounds a bit clunky.
> It's annoyingly slow to open
> a thread with 10,000 messages just to read one SMS. I'm almost tempted
> to mangle the messages on delivery and remove the References header
> before notmuch sees them, but it would be nice to have a cleaner
> solution, as there are other situations in which one might want to
> "reset" a really long thread.
Like this thread ;).
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Breaking a really long thread
2016-04-02 12:41 ` David Bremner
@ 2016-04-02 13:56 ` David Mazieres
2016-04-04 11:07 ` Eric
0 siblings, 1 reply; 38+ messages in thread
From: David Mazieres @ 2016-04-02 13:56 UTC (permalink / raw)
To: David Bremner, notmuch
David Bremner <david@tethera.net> writes:
> David Mazieres <dm-list-email-notmuch@scs.stanford.edu> writes:
>
>> Is there any way to break an existing thread (so as to start over with a
>> smaller thread), or otherwise to tweak the threading rules so that a
>> particular References header gets ignored.
>
> Currently there is no way to do this, as threads are "stateless"
> i.e. created on the fly by _notmuch_create_thread based only on
> immutable mail data.
Thanks.
>> It's annoyingly slow to open
>> a thread with 10,000 messages just to read one SMS. I'm almost tempted
>> to mangle the messages on delivery and remove the References header
>> before notmuch sees them, but it would be nice to have a cleaner
>> solution, as there are other situations in which one might want to
>> "reset" a really long thread.
>
> Like this thread ;).
Oops, sorry for the irrelevant thread inclusion. I guess emacs adds the
References header after a message is sent is sent? In my setup, the
easiest way to post to a mailing list is to reply to an existing message
(since I subscribe to each list under a different email address). I
tried to start a new thread by deleting the In-Reply-To and header which
was all I saw, but I guess the References header got inserted later...
David
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Breaking a really long thread
2016-04-02 13:56 ` David Mazieres
@ 2016-04-04 11:07 ` Eric
2016-04-04 13:00 ` Mark Walters
0 siblings, 1 reply; 38+ messages in thread
From: Eric @ 2016-04-04 11:07 UTC (permalink / raw)
To: notmuch
On Sat, 02 Apr 2016 06:56:12 -0700, David Mazieres <dm-list-email-notmuch@scs.stanford.edu> wrote:
> David Bremner <david@tethera.net> writes:
>
>> David Mazieres <dm-list-email-notmuch@scs.stanford.edu> writes:
>>
>>> Is there any way to break an existing thread (so as to start over with a
>>> smaller thread), or otherwise to tweak the threading rules so that a
>>> particular References header gets ignored.
>>
>> Currently there is no way to do this, as threads are "stateless"
>> i.e. created on the fly by _notmuch_create_thread based only on
>> immutable mail data.
>
> Thanks.
>
>>> It's annoyingly slow to open
>>> a thread with 10,000 messages just to read one SMS. I'm almost tempted
>>> to mangle the messages on delivery and remove the References header
>>> before notmuch sees them, but it would be nice to have a cleaner
>>> solution, as there are other situations in which one might want to
>>> "reset" a really long thread.
>>
>> Like this thread ;).
>
> Oops, sorry for the irrelevant thread inclusion. I guess emacs adds the
> References header after a message is sent is sent? In my setup, the
> easiest way to post to a mailing list is to reply to an existing message
> (since I subscribe to each list under a different email address). I
> tried to start a new thread by deleting the In-Reply-To and header which
> was all I saw, but I guess the References header got inserted later...
Neither notmuch emacs nor any other email client has any business
inserting a References header after the user "presses Send". On a new
message it shouldn't exist unless inserted manually, and on a reply it
should come from the replied-to message (and be changed) before you start
"replying". More likely that (if you didn't miss it) it was not shown
to you although it existed - that would count as a bug in my opinion
(I don't use emacs for anything, not even notmuch).
Actually the message you replied to has no References header, but notmuch
reply (command line) to it generates both References and In-Reply-To
(same content). I assume notmuch emacs does the same. I don't believe
that References should be generated in this situation, its only use
would be by a threading algorithm that doesn't use In-Reply-To, and I
would consider that a bug in said algorithm.
Actually I think there should be a "reply as new" option which uses
the other message but does not add either In-Reply-To or References
(and does not carry the latter forward if it exists).
Eric
--
ms fnd in a lbry
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Breaking a really long thread
2016-04-04 11:07 ` Eric
@ 2016-04-04 13:00 ` Mark Walters
2016-04-04 15:38 ` Eric
2016-04-05 5:28 ` David Mazieres
0 siblings, 2 replies; 38+ messages in thread
From: Mark Walters @ 2016-04-04 13:00 UTC (permalink / raw)
To: Eric, notmuch
On Mon, 04 Apr 2016, Eric <eric@deptj.eu> wrote:
> On Sat, 02 Apr 2016 06:56:12 -0700, David Mazieres <dm-list-email-notmuch@scs.stanford.edu> wrote:
>> David Bremner <david@tethera.net> writes:
>>
>>> David Mazieres <dm-list-email-notmuch@scs.stanford.edu> writes:
>>>
>>>> Is there any way to break an existing thread (so as to start over with a
>>>> smaller thread), or otherwise to tweak the threading rules so that a
>>>> particular References header gets ignored.
>>>
>>> Currently there is no way to do this, as threads are "stateless"
>>> i.e. created on the fly by _notmuch_create_thread based only on
>>> immutable mail data.
>>
>> Thanks.
>>
>>>> It's annoyingly slow to open
>>>> a thread with 10,000 messages just to read one SMS. I'm almost tempted
>>>> to mangle the messages on delivery and remove the References header
>>>> before notmuch sees them, but it would be nice to have a cleaner
>>>> solution, as there are other situations in which one might want to
>>>> "reset" a really long thread.
>>>
>>> Like this thread ;).
>>
>> Oops, sorry for the irrelevant thread inclusion. I guess emacs adds the
>> References header after a message is sent is sent? In my setup, the
>> easiest way to post to a mailing list is to reply to an existing message
>> (since I subscribe to each list under a different email address). I
>> tried to start a new thread by deleting the In-Reply-To and header which
>> was all I saw, but I guess the References header got inserted later...
>
> Neither notmuch emacs nor any other email client has any business
> inserting a References header after the user "presses Send". On a new
> message it shouldn't exist unless inserted manually, and on a reply it
> should come from the replied-to message (and be changed) before you start
> "replying". More likely that (if you didn't miss it) it was not shown
> to you although it existed - that would count as a bug in my opinion
> (I don't use emacs for anything, not even notmuch).
By default the reference header is hidden. It is controlled by
message-hidden-headers which you can customize. (Note notmuch adds
user-agent to this list via notmuch-mua-hidden-header.)
> Actually the message you replied to has no References header, but notmuch
> reply (command line) to it generates both References and In-Reply-To
> (same content). I assume notmuch emacs does the same. I don't believe
> that References should be generated in this situation, its only use
> would be by a threading algorithm that doesn't use In-Reply-To, and I
> would consider that a bug in said algorithm.
That isn't my reading of RFC2822 (section 3.6.4):
The "References:" field will contain the contents of the parent's
"References:" field (if any) followed by the contents of the parent's
"Message-ID:" field (if any).
> Actually I think there should be a "reply as new" option which uses
> the other message but does not add either In-Reply-To or References
> (and does not carry the latter forward if it exists).
That would be possible. If you don't actually want to include the
message itself then "c f" to stash the from, and then pasting that as
the "to" address gets pretty close.
Best wishes
Mark
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Breaking a really long thread
2016-04-04 13:00 ` Mark Walters
@ 2016-04-04 15:38 ` Eric
2016-04-05 5:28 ` David Mazieres
1 sibling, 0 replies; 38+ messages in thread
From: Eric @ 2016-04-04 15:38 UTC (permalink / raw)
To: notmuch
On Mon, 04 Apr 2016 14:00:26 +0100, Mark Walters <markwalters1009@gmail.com> wrote:
> On Mon, 04 Apr 2016, Eric <eric@deptj.eu> wrote:
>> On Sat, 02 Apr 2016 06:56:12 -0700, David Mazieres <dm-list-email-notmuch@scs.stanford.edu> wrote:
>>> David Bremner <david@tethera.net> writes:
>>>
>>>> David Mazieres <dm-list-email-notmuch@scs.stanford.edu> writes:
>>>>
>>>>> Is there any way to break an existing thread (so as to start over with a
>>>>> smaller thread), or otherwise to tweak the threading rules so that a
>>>>> particular References header gets ignored.
>>>>
>>>> Currently there is no way to do this, as threads are "stateless"
>>>> i.e. created on the fly by _notmuch_create_thread based only on
>>>> immutable mail data.
>>>
>>> Thanks.
>>>
>>>>> It's annoyingly slow to open
>>>>> a thread with 10,000 messages just to read one SMS. I'm almost tempted
>>>>> to mangle the messages on delivery and remove the References header
>>>>> before notmuch sees them, but it would be nice to have a cleaner
>>>>> solution, as there are other situations in which one might want to
>>>>> "reset" a really long thread.
>>>>
>>>> Like this thread ;).
>>>
>>> Oops, sorry for the irrelevant thread inclusion. I guess emacs adds the
>>> References header after a message is sent is sent? In my setup, the
>>> easiest way to post to a mailing list is to reply to an existing message
>>> (since I subscribe to each list under a different email address). I
>>> tried to start a new thread by deleting the In-Reply-To and header which
>>> was all I saw, but I guess the References header got inserted later...
>>
>> Neither notmuch emacs nor any other email client has any business
>> inserting a References header after the user "presses Send". On a new
>> message it shouldn't exist unless inserted manually, and on a reply it
>> should come from the replied-to message (and be changed) before you start
>> "replying". More likely that (if you didn't miss it) it was not shown
>> to you although it existed - that would count as a bug in my opinion
>> (I don't use emacs for anything, not even notmuch).
>
> By default the reference header is hidden. It is controlled by
> message-hidden-headers which you can customize. (Note notmuch adds
> user-agent to this list via notmuch-mua-hidden-header.)
Ah. Well of course I didn't know that since I don't use emacs. I guess
that if the OP is going to use reply to start new threads he should
unhide it.
>> Actually the message you replied to has no References header, but notmuch
>> reply (command line) to it generates both References and In-Reply-To
>> (same content). I assume notmuch emacs does the same. I don't believe
>> that References should be generated in this situation, its only use
>> would be by a threading algorithm that doesn't use In-Reply-To, and I
>> would consider that a bug in said algorithm.
>
> That isn't my reading of RFC2822 (section 3.6.4):
>
> The "References:" field will contain the contents of the parent's
> "References:" field (if any) followed by the contents of the parent's
> "Message-ID:" field (if any).
OK, I guess it should be there. In which case it shouldn't default to
hidden in any MUA, far too many people use the reply-for-new approach
without understanding it.
>> Actually I think there should be a "reply as new" option which uses
>> the other message but does not add either In-Reply-To or References
>> (and does not carry the latter forward if it exists).
>
> That would be possible. If you don't actually want to include the
> message itself then "c f" to stash the from, and then pasting that as
> the "to" address gets pretty close.
Eric
--
ms fnd in a lbry
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Breaking a really long thread
2016-04-04 13:00 ` Mark Walters
2016-04-04 15:38 ` Eric
@ 2016-04-05 5:28 ` David Mazieres
2016-04-09 11:20 ` Daniel Kahn Gillmor
1 sibling, 1 reply; 38+ messages in thread
From: David Mazieres @ 2016-04-05 5:28 UTC (permalink / raw)
To: Mark Walters, Eric, notmuch
Mark Walters <markwalters1009@gmail.com> writes:
> By default the reference header is hidden. It is controlled by
> message-hidden-headers which you can customize. (Note notmuch adds
> user-agent to this list via notmuch-mua-hidden-header.)
Thanks for explaining this! So I posted about one problem, and instead
got a solution to a problem I didn't even realize I had. Adding:
(setq message-hidden-headers (delete "^References:" message-hidden-headers))
to my eval-after-loaded notmuch-config.el file solved this problem
cold. No more unintentional References: headers for me.
Arguably, I would say either both the In-Reply-To and the References
header should be hidden or neither. Otherwise, what was happening is
that I was deleting the In-Reply-To header as it was the only one I saw,
and figuring that maybe References was adjusted after the fact based on
In-Reply-To. After all, the message buffer doesn't keep track of the
parent message.
Unless there's a reason that someone would want to alter In-Reply-To
without altering References, it doesn't make sense to show one without
the other.
David
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: Breaking a really long thread
2016-04-05 5:28 ` David Mazieres
@ 2016-04-09 11:20 ` Daniel Kahn Gillmor
2016-04-09 18:55 ` David Bremner
0 siblings, 1 reply; 38+ messages in thread
From: Daniel Kahn Gillmor @ 2016-04-09 11:20 UTC (permalink / raw)
To: David Mazieres expires 2016-07-03 PDT, Mark Walters, Eric,
notmuch
On Tue 2016-04-05 01:28:43 -0400, David Mazieres wrote:
> Arguably, I would say either both the In-Reply-To and the References
> header should be hidden or neither. Otherwise, what was happening is
> that I was deleting the In-Reply-To header as it was the only one I saw,
> and figuring that maybe References was adjusted after the fact based on
> In-Reply-To. After all, the message buffer doesn't keep track of the
> parent message.
>
> Unless there's a reason that someone would want to alter In-Reply-To
> without altering References, it doesn't make sense to show one without
> the other.
I think i agree with David here, but the fact is that
message-hidden-headers is derived directly from emacs (in message.el),
and isn't part of notmuch-emacs at all.
Are these changes worth addressing upstream?
--dkg
^ permalink raw reply [flat|nested] 38+ messages in thread* Re: Breaking a really long thread
2016-04-09 11:20 ` Daniel Kahn Gillmor
@ 2016-04-09 18:55 ` David Bremner
2016-04-09 22:40 ` Mark Walters
0 siblings, 1 reply; 38+ messages in thread
From: David Bremner @ 2016-04-09 18:55 UTC (permalink / raw)
To: Daniel Kahn Gillmor, David Mazieres expires 2016-07-03 PDT,
notmuch
Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:
> On Tue 2016-04-05 01:28:43 -0400, David Mazieres wrote:
>> Arguably, I would say either both the In-Reply-To and the References
>> header should be hidden or neither. Otherwise, what was happening is
>> that I was deleting the In-Reply-To header as it was the only one I saw,
>> and figuring that maybe References was adjusted after the fact based on
>> In-Reply-To. After all, the message buffer doesn't keep track of the
>> parent message.
>>
>> Unless there's a reason that someone would want to alter In-Reply-To
>> without altering References, it doesn't make sense to show one without
>> the other.
>
> I think i agree with David here, but the fact is that
> message-hidden-headers is derived directly from emacs (in message.el),
> and isn't part of notmuch-emacs at all.
>
> Are these changes worth addressing upstream?
Possibly. Although changing defaults is usually a cesspit of
bikeshedding. What would we ask, that upstream add In-Reply-To to
message-hidden-headers?
Related, showing hidden headers doesn't actually work very well:
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=23252
I thought briefly about overriding the value in notmuch-message-mode,
perhaps by having a defcustom for notmuch-message-hidden-headers.
d
^ permalink raw reply [flat|nested] 38+ messages in thread* Re: Breaking a really long thread
2016-04-09 18:55 ` David Bremner
@ 2016-04-09 22:40 ` Mark Walters
2016-04-11 2:05 ` David Bremner
0 siblings, 1 reply; 38+ messages in thread
From: Mark Walters @ 2016-04-09 22:40 UTC (permalink / raw)
To: David Bremner, Daniel Kahn Gillmor,
David Mazieres expires 2016-07-03 PDT, notmuch
Hi
On Sat, 09 Apr 2016, David Bremner <david@tethera.net> wrote:
> Daniel Kahn Gillmor <dkg@fifthhorseman.net> writes:
>
>> On Tue 2016-04-05 01:28:43 -0400, David Mazieres wrote:
>>> Arguably, I would say either both the In-Reply-To and the References
>>> header should be hidden or neither. Otherwise, what was happening is
>>> that I was deleting the In-Reply-To header as it was the only one I saw,
>>> and figuring that maybe References was adjusted after the fact based on
>>> In-Reply-To. After all, the message buffer doesn't keep track of the
>>> parent message.
>>>
>>> Unless there's a reason that someone would want to alter In-Reply-To
>>> without altering References, it doesn't make sense to show one without
>>> the other.
>>
>> I think i agree with David here, but the fact is that
>> message-hidden-headers is derived directly from emacs (in message.el),
>> and isn't part of notmuch-emacs at all.
>>
>> Are these changes worth addressing upstream?
>
> Possibly. Although changing defaults is usually a cesspit of
> bikeshedding. What would we ask, that upstream add In-Reply-To to
> message-hidden-headers?
>
> Related, showing hidden headers doesn't actually work very well:
>
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=23252
>
> I thought briefly about overriding the value in notmuch-message-mode,
> perhaps by having a defcustom for notmuch-message-hidden-headers.
I think we already have this, except it is called
notmuch-mua-hidden-headers. It defaults to '("^User-Agent:").
I think it would be reasonable to add In-Reply-To to this list.
However, if I read the code correctly, currently we are changing
message-hidden-headers globally which doesn't feel the right thing to
do. Probably we should do something more like you suggest, and do the
overriding just in notmuch-message-mode.
Best wishes
Mark
^ permalink raw reply [flat|nested] 38+ messages in thread* Re: Breaking a really long thread
2016-04-09 22:40 ` Mark Walters
@ 2016-04-11 2:05 ` David Bremner
2016-04-11 7:19 ` Mark Walters
0 siblings, 1 reply; 38+ messages in thread
From: David Bremner @ 2016-04-11 2:05 UTC (permalink / raw)
To: Mark Walters, Daniel Kahn Gillmor,
David Mazieres expires 2016-07-03 PDT, notmuch
Mark Walters <markwalters1009@gmail.com> writes:
> I think we already have this, except it is called
> notmuch-mua-hidden-headers. It defaults to '("^User-Agent:").
>
> I think it would be reasonable to add In-Reply-To to this list.
>
> However, if I read the code correctly, currently we are changing
> message-hidden-headers globally which doesn't feel the right thing to
> do. Probably we should do something more like you suggest, and do the
> overriding just in notmuch-message-mode.
>
Summarizing, I see 3 related changes here
- fixing the code to let-bind message-hidden-headers instead of globally
modify it
- adding in-reply-to notmuch-mua
- providing a keybinding to toggle visibility of hidden headers that
works better than the current use of widen
d
^ permalink raw reply [flat|nested] 38+ messages in thread* Re: Breaking a really long thread
2016-04-11 2:05 ` David Bremner
@ 2016-04-11 7:19 ` Mark Walters
2016-04-11 7:39 ` David Edmondson
0 siblings, 1 reply; 38+ messages in thread
From: Mark Walters @ 2016-04-11 7:19 UTC (permalink / raw)
To: David Bremner, Daniel Kahn Gillmor,
David Mazieres expires 2016-07-03 PDT, notmuch
On Mon, 11 Apr 2016, David Bremner <david@tethera.net> wrote:
> Mark Walters <markwalters1009@gmail.com> writes:
>
>> I think we already have this, except it is called
>> notmuch-mua-hidden-headers. It defaults to '("^User-Agent:").
>>
>> I think it would be reasonable to add In-Reply-To to this list.
>>
>> However, if I read the code correctly, currently we are changing
>> message-hidden-headers globally which doesn't feel the right thing to
>> do. Probably we should do something more like you suggest, and do the
>> overriding just in notmuch-message-mode.
>>
>
> Summarizing, I see 3 related changes here
>
> - fixing the code to let-bind message-hidden-headers instead of globally
> modify it
I have a patch doing this, but there is one problem that i don't see how
to solve neatly. At the moment we set hidden-headers to the union of
message-hidden-headers and notmuch-mua-hidden-headers. This means that
if the user wants to show a header that is normally one of
message-hidden-headers they have to set message-hidden-headers (ie they
can't just change notmuch-mua-hidden headers).
If we want it that people can show such a header just by changing a
notmuch variable then I don't see how to do it neatly without breaking
some existing setups.
Any suggestions?
Best wishes
Mark
>
> - adding in-reply-to notmuch-mua
>
> - providing a keybinding to toggle visibility of hidden headers that
> works better than the current use of widen
>
> d
^ permalink raw reply [flat|nested] 38+ messages in thread* Re: Breaking a really long thread
2016-04-11 7:19 ` Mark Walters
@ 2016-04-11 7:39 ` David Edmondson
2016-04-11 9:57 ` David Bremner
0 siblings, 1 reply; 38+ messages in thread
From: David Edmondson @ 2016-04-11 7:39 UTC (permalink / raw)
To: Mark Walters, David Bremner, Daniel Kahn Gillmor,
David Mazieres expires 2016-07-03 PDT, notmuch
On Mon, Apr 11 2016, Mark Walters wrote:
> On Mon, 11 Apr 2016, David Bremner <david@tethera.net> wrote:
>> Mark Walters <markwalters1009@gmail.com> writes:
>>
>>> I think we already have this, except it is called
>>> notmuch-mua-hidden-headers. It defaults to '("^User-Agent:").
>>>
>>> I think it would be reasonable to add In-Reply-To to this list.
>>>
>>> However, if I read the code correctly, currently we are changing
>>> message-hidden-headers globally which doesn't feel the right thing to
>>> do. Probably we should do something more like you suggest, and do the
>>> overriding just in notmuch-message-mode.
>>>
>>
>> Summarizing, I see 3 related changes here
>>
>> - fixing the code to let-bind message-hidden-headers instead of globally
>> modify it
>
> I have a patch doing this, but there is one problem that i don't see how
> to solve neatly. At the moment we set hidden-headers to the union of
> message-hidden-headers and notmuch-mua-hidden-headers. This means that
> if the user wants to show a header that is normally one of
> message-hidden-headers they have to set message-hidden-headers (ie they
> can't just change notmuch-mua-hidden headers).
>
> If we want it that people can show such a header just by changing a
> notmuch variable then I don't see how to do it neatly without breaking
> some existing setups.
>
> Any suggestions?
My inclination is to say "if you want to change the headers that are
hidden in message mode, change `message-hidden-headers'" and don't
bother with any notmuch specific settings.
That, of course, leads me to wonder why I ever added
`notmuch-mua-hidden-headers' in the first place...
> Best wishes
>
> Mark
>
>>
>> - adding in-reply-to notmuch-mua
>>
>> - providing a keybinding to toggle visibility of hidden headers that
>> works better than the current use of widen
>>
>> d
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> https://notmuchmail.org/mailman/listinfo/notmuch
^ permalink raw reply [flat|nested] 38+ messages in thread* Re: Breaking a really long thread
2016-04-11 7:39 ` David Edmondson
@ 2016-04-11 9:57 ` David Bremner
0 siblings, 0 replies; 38+ messages in thread
From: David Bremner @ 2016-04-11 9:57 UTC (permalink / raw)
To: David Edmondson, Mark Walters, Daniel Kahn Gillmor,
David Mazieres expires 2016-07-03 PDT, notmuch
David Edmondson <dme@dme.org> writes:
>
> My inclination is to say "if you want to change the headers that are
> hidden in message mode, change `message-hidden-headers'" and don't
> bother with any notmuch specific settings.
>
> That, of course, leads me to wonder why I ever added
> `notmuch-mua-hidden-headers' in the first place...
Whatever we decide to do, I'd like to avoid notmuch making persistent
changes to message-mode settings, to avoid problems for people who
e.g. use notmuch as an indexing backend for gnus.
d
^ permalink raw reply [flat|nested] 38+ messages in thread