* library config API
@ 2016-01-23 14:59 David Bremner
2016-01-23 14:59 ` [PATCH 1/5] lib: provide " David Bremner
` (5 more replies)
0 siblings, 6 replies; 15+ messages in thread
From: David Bremner @ 2016-01-23 14:59 UTC (permalink / raw)
To: notmuch
This is the third revision of this series, and I think the first two
patches are ready for serious review.
Some things to think about with respect to the CLI patches
- Should we include a "version" line at the top of the dump? Currently this is unused, but that
depends on the format being auto detectable?
- Should config info be included by default in the dump output?
- Should we restore config info by default?
- How about the usual bikeshedding about argument names? I'm loath to
spend time on docs until after this.
I guess a global question is what do we need this API / feature for. I
can think of three things so far
- tag aliases [1]
- making the location of the mail tree configurable (i.e. allowing
mailtree and database to live in different places).
- Some things like excludes and maildir synchonization are natural to
have in common between all clients.
A natural question is why not provide a library API to access
.notmuch-config? Having a plain text representation of the
configuration (without the extra step of dumping it) is obviously
attractive, but
- We would need to build some locking / synchronization to prevent
various clients and interactive text editors from stomping on each
other. This we have for free by using xapian metadata.
- We'd also have to provide our own key/value datastructure.
- At least for tag aliases, I think there is a second argument that
this information part of the users tagging state, and should be
backed up as part of the normal dump / restore process.
[1]: id:87mvsd7cxr.fsf@zancas.localnet
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH 1/5] lib: provide config API
2016-01-23 14:59 library config API David Bremner
@ 2016-01-23 14:59 ` David Bremner
2016-01-23 14:59 ` [PATCH 2/5] lib: config list iterators David Bremner
` (4 subsequent siblings)
5 siblings, 0 replies; 15+ messages in thread
From: David Bremner @ 2016-01-23 14:59 UTC (permalink / raw)
To: notmuch
This is a thin wrapper around the Xapian metadata API. The job of this
layer is to keep the config key value pairs from colliding with other
metadata by transparently prefixing the keys, along with the usual glue
to provide a C interface.
The split of _get_config into two functions is to allow returning of the
return value with different memory ownership semantics.
---
lib/Makefile.local | 1 +
lib/config.cc | 90 ++++++++++++++++++++++++++++++++++++++++++++++++++
lib/notmuch.h | 20 +++++++++++
test/T590-libconfig.sh | 58 ++++++++++++++++++++++++++++++++
4 files changed, 169 insertions(+)
create mode 100644 lib/config.cc
create mode 100755 test/T590-libconfig.sh
diff --git a/lib/Makefile.local b/lib/Makefile.local
index 3a07090..eb442d1 100644
--- a/lib/Makefile.local
+++ b/lib/Makefile.local
@@ -48,6 +48,7 @@ libnotmuch_cxx_srcs = \
$(dir)/index.cc \
$(dir)/message.cc \
$(dir)/query.cc \
+ $(dir)/config.cc \
$(dir)/thread.cc
libnotmuch_modules := $(libnotmuch_c_srcs:.c=.o) $(libnotmuch_cxx_srcs:.cc=.o)
diff --git a/lib/config.cc b/lib/config.cc
new file mode 100644
index 0000000..af00d6f
--- /dev/null
+++ b/lib/config.cc
@@ -0,0 +1,90 @@
+/* metadata.cc - API for database metadata
+ *
+ * Copyright © 2015 David Bremner
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see http://www.gnu.org/licenses/ .
+ *
+ * Author: David Bremner <david@tethera.net>
+ */
+
+#include "notmuch.h"
+#include "notmuch-private.h"
+#include "database-private.h"
+
+static const std::string CONFIG_PREFIX="C";
+
+notmuch_status_t
+notmuch_database_set_config (notmuch_database_t *notmuch,
+ const char *key,
+ const char *value)
+{
+ notmuch_status_t status;
+ Xapian::WritableDatabase *db;
+
+ status = _notmuch_database_ensure_writable (notmuch);
+ if (status)
+ return status;
+
+ try {
+ db = static_cast <Xapian::WritableDatabase *> (notmuch->xapian_db);
+ db->set_metadata (CONFIG_PREFIX+key, value);
+ } catch (const Xapian::Error &error) {
+ status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
+ notmuch->exception_reported = TRUE;
+ if (! notmuch->exception_reported) {
+ _notmuch_database_log (notmuch, "Error: A Xapian exception occurred setting metadata: %s\n",
+ error.get_msg().c_str());
+ }
+ }
+ return NOTMUCH_STATUS_SUCCESS;
+}
+
+static notmuch_status_t
+_metadata_value (notmuch_database_t *notmuch,
+ const char *key,
+ std::string &value)
+{
+ notmuch_status_t status = NOTMUCH_STATUS_SUCCESS;
+
+ try {
+ value = notmuch->xapian_db->get_metadata (CONFIG_PREFIX+key);
+ } catch (const Xapian::Error &error) {
+ status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
+ notmuch->exception_reported = TRUE;
+ if (! notmuch->exception_reported) {
+ _notmuch_database_log (notmuch, "Error: A Xapian exception occurred getting metadata: %s\n",
+ error.get_msg().c_str());
+ }
+ }
+ return status;
+}
+
+notmuch_status_t
+notmuch_database_get_config (notmuch_database_t *notmuch,
+ const char *key,
+ char **value) {
+ std::string strval;
+ notmuch_status_t status;
+
+ if (!value)
+ return NOTMUCH_STATUS_NULL_POINTER;
+
+ status = _metadata_value (notmuch, key, strval);
+ if (status)
+ return status;
+
+ *value = strdup (strval.c_str ());
+
+ return NOTMUCH_STATUS_SUCCESS;
+}
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 310a8b8..c62223b 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -1829,6 +1829,26 @@ notmuch_filenames_move_to_next (notmuch_filenames_t *filenames);
void
notmuch_filenames_destroy (notmuch_filenames_t *filenames);
+
+/**
+ * set config 'key' to 'value'
+ *
+ */
+notmuch_status_t
+notmuch_database_set_config (notmuch_database_t *db, const char *key, const char *value);
+
+/**
+ * retrieve config item 'key', assign to 'value'
+ *
+ * keys which have not been previously set with n_d_set_config will
+ * return an empty string.
+ *
+ * return value is allocated by malloc and should be freed by the
+ * caller.
+ */
+notmuch_status_t
+notmuch_database_get_config (notmuch_database_t *db, const char *key, char **value);
+
/* @} */
NOTMUCH_END_DECLS
diff --git a/test/T590-libconfig.sh b/test/T590-libconfig.sh
new file mode 100755
index 0000000..85e4497
--- /dev/null
+++ b/test/T590-libconfig.sh
@@ -0,0 +1,58 @@
+#!/usr/bin/env bash
+test_description="library config API"
+
+. ./test-lib.sh || exit 1
+
+add_email_corpus
+
+cat <<EOF > c_head
+#include <stdio.h>
+#include <string.h>
+#include <stdlib.h>
+#include <notmuch.h>
+
+void run(int line, notmuch_status_t ret)
+{
+ if (ret) {
+ fprintf (stderr, "line %d: %s\n", line, ret);
+ exit (1);
+ }
+}
+
+#define RUN(v) run(__LINE__, v);
+
+int main (int argc, char** argv)
+{
+ notmuch_database_t *db;
+ char *val;
+ notmuch_status_t stat;
+
+ RUN(notmuch_database_open (argv[1], NOTMUCH_DATABASE_MODE_READ_WRITE, &db));
+
+EOF
+
+cat <<EOF > c_tail
+ RUN(notmuch_database_destroy(db));
+}
+EOF
+
+test_begin_subtest "notmuch_database_{set,get}_config"
+cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR}
+{
+ RUN(notmuch_database_set_config (db, "testkey1", "testvalue1"));
+ RUN(notmuch_database_set_config (db, "testkey2", "testvalue2"));
+ RUN(notmuch_database_get_config (db, "testkey1", &val));
+ printf("testkey1 = %s\n", val);
+ RUN(notmuch_database_get_config (db, "testkey2", &val));
+ printf("testkey2 = %s\n", val);
+}
+EOF
+cat <<'EOF' >EXPECTED
+== stdout ==
+testkey1 = testvalue1
+testkey2 = testvalue2
+== stderr ==
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_done
--
2.6.4
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 2/5] lib: config list iterators
2016-01-23 14:59 library config API David Bremner
2016-01-23 14:59 ` [PATCH 1/5] lib: provide " David Bremner
@ 2016-01-23 14:59 ` David Bremner
2016-01-23 14:59 ` [PATCH 3/5] CLI: add print_status_database David Bremner
` (3 subsequent siblings)
5 siblings, 0 replies; 15+ messages in thread
From: David Bremner @ 2016-01-23 14:59 UTC (permalink / raw)
To: notmuch
Since xapian provides the ability to restrict the iterator to a given
prefix, we expose this ability to the user. Otherwise we mimic the other
iterator interfances in notmuch (e.g. tags.c).
---
lib/config.cc | 104 +++++++++++++++++++++++++++++++++++++++++++++++++
lib/notmuch.h | 44 +++++++++++++++++++++
test/T590-libconfig.sh | 60 ++++++++++++++++++++++++++++
3 files changed, 208 insertions(+)
diff --git a/lib/config.cc b/lib/config.cc
index af00d6f..e581f32 100644
--- a/lib/config.cc
+++ b/lib/config.cc
@@ -24,6 +24,19 @@
static const std::string CONFIG_PREFIX="C";
+struct _notmuch_config_list {
+ notmuch_database_t *notmuch;
+ Xapian::TermIterator *iterator;
+ char *current_key;
+ char *current_val;
+};
+
+static int
+_notmuch_config_list_destroy (notmuch_config_list_t *list) {
+ delete list->iterator;
+ return 0;
+}
+
notmuch_status_t
notmuch_database_set_config (notmuch_database_t *notmuch,
const char *key,
@@ -88,3 +101,94 @@ notmuch_database_get_config (notmuch_database_t *notmuch,
return NOTMUCH_STATUS_SUCCESS;
}
+
+notmuch_status_t
+notmuch_database_get_config_list (notmuch_database_t *notmuch,
+ const char *prefix,
+ notmuch_config_list_t **out)
+{
+ notmuch_config_list_t *list = NULL;
+ notmuch_status_t status = NOTMUCH_STATUS_SUCCESS;
+
+ list = talloc (notmuch, notmuch_config_list_t);
+ if (!list) {
+ status = NOTMUCH_STATUS_OUT_OF_MEMORY;
+ goto DONE;
+ }
+
+ talloc_set_destructor (list, _notmuch_config_list_destroy);
+ list->iterator = new Xapian::TermIterator;
+ list->notmuch = notmuch;
+ list->current_key = NULL;
+ list->current_val = NULL;
+
+ try {
+
+ *list->iterator = notmuch->xapian_db->metadata_keys_begin (CONFIG_PREFIX + (prefix ? prefix : ""));
+
+ } catch (const Xapian::Error &error) {
+ _notmuch_database_log (notmuch, "A Xapian exception occurred getting metadata iterator: %s.\n",
+ error.get_msg().c_str());
+ notmuch->exception_reported = TRUE;
+ status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
+ }
+
+ *out = list;
+
+ DONE:
+ if (status && list)
+ talloc_free (list);
+
+ return status;
+}
+
+notmuch_bool_t
+notmuch_config_list_valid (notmuch_config_list_t *metadata)
+{
+ if (*(metadata->iterator) == metadata->notmuch->xapian_db->metadata_keys_end())
+ return FALSE;
+
+ return TRUE;
+}
+
+const char *
+notmuch_config_list_key (notmuch_config_list_t *list)
+{
+ if (list->current_key)
+ talloc_free (list->current_key);
+
+ list->current_key = talloc_strdup (list, (**(list->iterator)).c_str () + CONFIG_PREFIX.length ());
+
+ return list->current_key;
+}
+
+const char *
+notmuch_config_list_value (notmuch_config_list_t *list)
+{
+ std::string strval;
+ notmuch_status_t status;
+ const char *key = notmuch_config_list_key (list);
+
+ /* TODO: better error reporting?? */
+ status = _metadata_value (list->notmuch, key, strval);
+ if (status)
+ return NULL;
+
+ if (list->current_val)
+ talloc_free(list->current_val);
+
+ list->current_val = talloc_strdup(list, strval.c_str ());
+ return list->current_val;
+}
+
+void
+notmuch_config_list_move_to_next (notmuch_config_list_t *list)
+{
+ (*(list->iterator))++;
+}
+
+void
+notmuch_config_list_destroy (notmuch_config_list_t *list)
+{
+ talloc_free (list);
+}
diff --git a/lib/notmuch.h b/lib/notmuch.h
index c62223b..b439c88 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -197,6 +197,7 @@ typedef struct _notmuch_message notmuch_message_t;
typedef struct _notmuch_tags notmuch_tags_t;
typedef struct _notmuch_directory notmuch_directory_t;
typedef struct _notmuch_filenames notmuch_filenames_t;
+typedef struct _notmuch_config_list notmuch_config_list_t;
#endif /* __DOXYGEN__ */
/**
@@ -1849,6 +1850,49 @@ notmuch_database_set_config (notmuch_database_t *db, const char *key, const char
notmuch_status_t
notmuch_database_get_config (notmuch_database_t *db, const char *key, char **value);
+/**
+ * Create an iterator for all config items with keys matching a given prefix
+ */
+notmuch_status_t
+notmuch_database_get_config_list (notmuch_database_t *db, const char *prefix, notmuch_config_list_t **out);
+
+/**
+ * Is 'config_list' iterator valid (i.e. _key, _value, _move_to_next can be called).
+ */
+notmuch_bool_t
+notmuch_config_list_valid (notmuch_config_list_t *config_list);
+
+/**
+ * return key for current config pair
+ *
+ * return value is owned by the iterator, and will be destroyed by the
+ * next call to notmuch_config_list_key or notmuch_config_list_destroy.
+ */
+const char *
+notmuch_config_list_key (notmuch_config_list_t *config_list);
+
+/**
+ * return 'value' for current config pair
+ *
+ * return value is owned by the iterator, and will be destroyed by the
+ * next call to notmuch_config_list_value or notmuch config_list_destroy
+ */
+const char *
+notmuch_config_list_value (notmuch_config_list_t *config_list);
+
+
+/**
+ * move 'config_list' iterator to the next pair
+ */
+void
+notmuch_config_list_move_to_next (notmuch_config_list_t *config_list);
+
+/**
+ * free any resources held by 'config_list'
+ */
+void
+notmuch_config_list_destroy (notmuch_config_list_t *config_list);
+
/* @} */
NOTMUCH_END_DECLS
diff --git a/test/T590-libconfig.sh b/test/T590-libconfig.sh
index 85e4497..8ca6883 100755
--- a/test/T590-libconfig.sh
+++ b/test/T590-libconfig.sh
@@ -55,4 +55,64 @@ testkey2 = testvalue2
EOF
test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "notmuch_database_get_config_list: empty list"
+cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR}
+{
+ notmuch_config_list_t *list;
+ RUN(notmuch_database_get_config_list (db, "nonexistent", &list));
+ printf("valid = %d\n", notmuch_config_list_valid (list));
+ notmuch_config_list_destroy (list);
+}
+EOF
+cat <<'EOF' >EXPECTED
+== stdout ==
+valid = 0
+== stderr ==
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+
+test_begin_subtest "notmuch_database_get_config_list: all pairs"
+cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR}
+{
+ notmuch_config_list_t *list;
+ RUN(notmuch_database_set_config (db, "zzzafter", "afterval"));
+ RUN(notmuch_database_set_config (db, "aaabefore", "beforeval"));
+ RUN(notmuch_database_get_config_list (db, "", &list));
+ for (; notmuch_config_list_valid (list); notmuch_config_list_move_to_next (list)) {
+ printf("%s %s\n", notmuch_config_list_key (list), notmuch_config_list_value(list));
+ }
+ notmuch_config_list_destroy (list);
+}
+EOF
+cat <<'EOF' >EXPECTED
+== stdout ==
+aaabefore beforeval
+testkey1 testvalue1
+testkey2 testvalue2
+zzzafter afterval
+== stderr ==
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "notmuch_database_get_config_list: one prefix"
+cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR}
+{
+ notmuch_config_list_t *list;
+ RUN(notmuch_database_get_config_list (db, "testkey", &list));
+ for (; notmuch_config_list_valid (list); notmuch_config_list_move_to_next (list)) {
+ printf("%s %s\n", notmuch_config_list_key (list), notmuch_config_list_value(list));
+ }
+ notmuch_config_list_destroy (list);
+}
+EOF
+cat <<'EOF' >EXPECTED
+== stdout ==
+testkey1 testvalue1
+testkey2 testvalue2
+== stderr ==
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
test_done
--
2.6.4
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 3/5] CLI: add print_status_database
2016-01-23 14:59 library config API David Bremner
2016-01-23 14:59 ` [PATCH 1/5] lib: provide " David Bremner
2016-01-23 14:59 ` [PATCH 2/5] lib: config list iterators David Bremner
@ 2016-01-23 14:59 ` David Bremner
2016-01-23 14:59 ` [PATCH 4/5] CLI: add optional config data to dump output David Bremner
` (2 subsequent siblings)
5 siblings, 0 replies; 15+ messages in thread
From: David Bremner @ 2016-01-23 14:59 UTC (permalink / raw)
To: notmuch
This could probably be used at quite a few places in the existing code,
but in the immediate future I plan to use in some new code in
notmuch-dump
---
notmuch-client.h | 5 +++++
status.c | 17 +++++++++++++++++
2 files changed, 22 insertions(+)
diff --git a/notmuch-client.h b/notmuch-client.h
index 3bd2903..7c9a1ea 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -459,6 +459,11 @@ print_status_query (const char *loc,
const notmuch_query_t *query,
notmuch_status_t status);
+notmuch_status_t
+print_status_database (const char *loc,
+ const notmuch_database_t *database,
+ notmuch_status_t status);
+
#include "command-line-arguments.h"
extern char *notmuch_requested_db_uuid;
diff --git a/status.c b/status.c
index 8fa81cb..45d3fb4 100644
--- a/status.c
+++ b/status.c
@@ -19,3 +19,20 @@ print_status_query (const char *loc,
}
return status;
}
+
+notmuch_status_t
+print_status_database (const char *loc,
+ const notmuch_database_t *notmuch,
+ notmuch_status_t status)
+{
+ if (status) {
+ const char *msg;
+
+ fprintf (stderr, "%s: %s\n", loc,
+ notmuch_status_to_string (status));
+ msg = notmuch_database_status_string (notmuch);
+ if (msg)
+ fputs (msg, stderr);
+ }
+ return status;
+}
--
2.6.4
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 4/5] CLI: add optional config data to dump output.
2016-01-23 14:59 library config API David Bremner
` (2 preceding siblings ...)
2016-01-23 14:59 ` [PATCH 3/5] CLI: add print_status_database David Bremner
@ 2016-01-23 14:59 ` David Bremner
2016-01-23 14:59 ` [PATCH 5/5] CLI: optionally restore config data David Bremner
2016-03-12 12:31 ` library config api v4 David Bremner
5 siblings, 0 replies; 15+ messages in thread
From: David Bremner @ 2016-01-23 14:59 UTC (permalink / raw)
To: notmuch
This lacks at least documentation. Note that it changes the default dump
output format, but doesn't break existing notmuch-restore. It might
break user scripts though.
---
notmuch-client.h | 8 +++++
notmuch-dump.c | 80 +++++++++++++++++++++++++++++++++++++++++++++--
notmuch-new.c | 2 +-
test/T150-tagging.sh | 8 ++---
test/T240-dump-restore.sh | 14 ++++-----
test/T590-libconfig.sh | 17 ++++++++++
test/test-lib.sh | 6 ++++
7 files changed, 120 insertions(+), 15 deletions(-)
diff --git a/notmuch-client.h b/notmuch-client.h
index 7c9a1ea..7588734 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -443,11 +443,19 @@ typedef enum dump_formats {
DUMP_FORMAT_SUP
} dump_format_t;
+typedef enum dump_includes {
+ DUMP_INCLUDE_TAGS=1,
+ DUMP_INCLUDE_CONFIG=2,
+} dump_include_t;
+
+#define NOTMUCH_DUMP_VERSION 2
+
int
notmuch_database_dump (notmuch_database_t *notmuch,
const char *output_file_name,
const char *query_str,
dump_format_t output_format,
+ dump_include_t include,
notmuch_bool_t gzip_output);
/* If status is non-zero (i.e. error) print appropriate
diff --git a/notmuch-dump.c b/notmuch-dump.c
index 829781f..a6cf810 100644
--- a/notmuch-dump.c
+++ b/notmuch-dump.c
@@ -23,16 +23,82 @@
#include "string-util.h"
#include <zlib.h>
+static int
+database_dump_config (notmuch_database_t *notmuch, gzFile output)
+{
+ notmuch_config_list_t *list;
+ int ret = EXIT_FAILURE;
+ char *buffer = NULL;
+ size_t buffer_size = 0;
+
+ if (print_status_database ("notmuch dump", notmuch,
+ notmuch_database_get_config_list (notmuch, NULL, &list)))
+ goto DONE;
+
+ for (; notmuch_config_list_valid (list); notmuch_config_list_move_to_next (list)) {
+ if (hex_encode (notmuch, notmuch_config_list_key (list),
+ &buffer, &buffer_size) != HEX_SUCCESS) {
+ fprintf (stderr, "Error: failed to hex-encode config key %s\n",
+ notmuch_config_list_key (list));
+ goto DONE;
+ }
+ gzprintf (output, "#@ %s", buffer);
+
+ if (hex_encode (notmuch, notmuch_config_list_value (list),
+ &buffer, &buffer_size) != HEX_SUCCESS) {
+ fprintf (stderr, "Error: failed to hex-encode config value %s\n",
+ notmuch_config_list_value (list) );
+ goto DONE;
+ }
+
+ gzprintf (output, " %s\n", buffer);
+ }
+
+ ret = EXIT_SUCCESS;
+
+ DONE:
+ if (list)
+ notmuch_config_list_destroy (list);
+
+ if (buffer)
+ talloc_free (buffer);
+
+ return ret;
+}
+
+static void
+print_dump_header (gzFile output, int output_format, int include)
+{
+ gzprintf (output, "#notmuch-dump %s:%d %s%s%s\n",
+ (output_format == DUMP_FORMAT_SUP) ? "sup" : "batch-tag",
+ NOTMUCH_DUMP_VERSION,
+ (include & DUMP_INCLUDE_CONFIG) ? "config" : "",
+ (include & DUMP_INCLUDE_TAGS) && (include & DUMP_INCLUDE_CONFIG) ? "," : "",
+ (include & DUMP_INCLUDE_TAGS) ? "tags" : "");
+
+
+}
static int
database_dump_file (notmuch_database_t *notmuch, gzFile output,
- const char *query_str, int output_format)
+ const char *query_str, int output_format, int include)
{
notmuch_query_t *query;
notmuch_messages_t *messages;
notmuch_message_t *message;
notmuch_tags_t *tags;
+ print_dump_header (output, output_format, include);
+
+ if (include & DUMP_INCLUDE_CONFIG) {
+ if (print_status_database ("notmuch dump", notmuch,
+ database_dump_config(notmuch,output)))
+ return EXIT_FAILURE;
+ }
+
+ if (! (include & DUMP_INCLUDE_TAGS))
+ return EXIT_SUCCESS;
+
if (! query_str)
query_str = "";
@@ -130,6 +196,7 @@ notmuch_database_dump (notmuch_database_t *notmuch,
const char *output_file_name,
const char *query_str,
dump_format_t output_format,
+ dump_include_t include,
notmuch_bool_t gzip_output)
{
gzFile output = NULL;
@@ -164,7 +231,7 @@ notmuch_database_dump (notmuch_database_t *notmuch,
goto DONE;
}
- ret = database_dump_file (notmuch, output, query_str, output_format);
+ ret = database_dump_file (notmuch, output, query_str, output_format, include);
if (ret) goto DONE;
ret = gzflush (output, Z_FINISH);
@@ -226,6 +293,7 @@ notmuch_dump_command (notmuch_config_t *config, int argc, char *argv[])
int opt_index;
int output_format = DUMP_FORMAT_BATCH_TAG;
+ int include = 0;
notmuch_bool_t gzip_output = 0;
notmuch_opt_desc_t options[] = {
@@ -233,6 +301,9 @@ notmuch_dump_command (notmuch_config_t *config, int argc, char *argv[])
(notmuch_keyword_t []){ { "sup", DUMP_FORMAT_SUP },
{ "batch-tag", DUMP_FORMAT_BATCH_TAG },
{ 0, 0 } } },
+ { NOTMUCH_OPT_KEYWORD_FLAGS, &include, "include", 'I',
+ (notmuch_keyword_t []){ { "config", DUMP_INCLUDE_CONFIG },
+ { "tags", DUMP_INCLUDE_TAGS} } },
{ NOTMUCH_OPT_STRING, &output_file_name, "output", 'o', 0 },
{ NOTMUCH_OPT_BOOLEAN, &gzip_output, "gzip", 'z', 0 },
{ NOTMUCH_OPT_INHERIT, (void *) ¬much_shared_options, NULL, 0, 0 },
@@ -245,6 +316,9 @@ notmuch_dump_command (notmuch_config_t *config, int argc, char *argv[])
notmuch_process_shared_options (argv[0]);
+ if (include == 0)
+ include = DUMP_INCLUDE_CONFIG | DUMP_INCLUDE_TAGS;
+
if (opt_index < argc) {
query_str = query_string_from_args (notmuch, argc - opt_index, argv + opt_index);
if (query_str == NULL) {
@@ -254,7 +328,7 @@ notmuch_dump_command (notmuch_config_t *config, int argc, char *argv[])
}
ret = notmuch_database_dump (notmuch, output_file_name, query_str,
- output_format, gzip_output);
+ output_format, include, gzip_output);
notmuch_database_destroy (notmuch);
diff --git a/notmuch-new.c b/notmuch-new.c
index e503776..fd2ff82 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -1041,7 +1041,7 @@ notmuch_new_command (notmuch_config_t *config, int argc, char *argv[])
}
if (notmuch_database_dump (notmuch, backup_name, "",
- DUMP_FORMAT_BATCH_TAG, TRUE)) {
+ DUMP_FORMAT_BATCH_TAG, DUMP_INCLUDE_CONFIG | DUMP_INCLUDE_TAGS, TRUE)) {
fprintf (stderr, "Backup failed. Aborting upgrade.");
return EXIT_FAILURE;
}
diff --git a/test/T150-tagging.sh b/test/T150-tagging.sh
index 8adcabc..f1a6df0 100755
--- a/test/T150-tagging.sh
+++ b/test/T150-tagging.sh
@@ -188,7 +188,7 @@ cat <<EOF > EXPECTED
+%22%27%22%27%22%22%27%27 +inbox +tag4 +tag5 +unread -- id:msg-002@notmuch-test-suite
EOF
-notmuch dump --format=batch-tag | sort > OUTPUT
+NOTMUCH_DUMP_TAGS > OUTPUT
notmuch restore --format=batch-tag < BACKUP
test_expect_equal_file EXPECTED OUTPUT
@@ -209,7 +209,7 @@ cat <<EOF > EXPECTED
+%21@%23%20%24%25%5e%26%2a%29-_=+%5b%7b%5c%20%7c%3b%3a%27%20%22,.%3c%60%7e +inbox +tag5 +unread -- id:msg-001@notmuch-test-suite
EOF
-notmuch dump --format=batch-tag | sort > OUTPUT
+NOTMUCH_DUMP_TAGS > OUTPUT
notmuch restore --format=batch-tag < BACKUP
test_expect_equal_file EXPECTED OUTPUT
@@ -235,7 +235,7 @@ cat <<EOF > EXPECTED
+%2a@%7d%cf%b5%f4%85%80%adO3%da%a7 +=%e0%ac%95%c8%b3+%ef%aa%95%c8%a64w%c7%9d%c9%a2%cf%b3%d6%82%24B%c4%a9%c5%a1UX%ee%99%b0%27E7%ca%a4%d0%8b%5d +A%e1%a0%bc%de%8b%d5%b2V%d9%9b%f3%b5%a2%a3M%d8%a1u@%f0%a0%ac%948%7e%f0%ab%86%af%27 +L%df%85%ef%a1%a5m@%d3%96%c2%ab%d4%9f%ca%b8%f3%b3%a2%bf%c7%b1_u%d7%b4%c7%b1 +P%c4%98%2f +R +inbox +tag5 +unread +%7e%d1%8b%25%ec%a0%ae%d1%a0M%3b%e3%b6%b7%e9%a4%87%3c%db%9a%cc%a8%e1%96%9d +%c4%bf7%c7%ab9H%c4%99k%ea%91%bd%c3%8ck%e2%b3%8dk%c5%952V%e4%99%b2%d9%b3%e4%8b%bda%5b%24%c7%9b +%da%88=f%cc%b9I%ce%af%7b%c9%97%e3%b9%8bH%cb%92X%d2%8c6 +%dc%9crh%d2%86B%e5%97%a2%22t%ed%99%82d -- id:msg-001@notmuch-test-suite
EOF
-notmuch dump --format=batch-tag | sort > OUTPUT
+NOTMUCH_DUMP_TAGS > OUTPUT
notmuch restore --format=batch-tag < BACKUP
test_expect_equal_file EXPECTED OUTPUT
@@ -260,7 +260,7 @@ cat <<EOF > EXPECTED
+foo%3a%3abar%25 +found%3a%3ait +inbox +tag5 +unread +winner -- id:msg-001@notmuch-test-suite
EOF
-notmuch dump --format=batch-tag | sort > OUTPUT
+NOTMUCH_DUMP_TAGS > OUTPUT
notmuch restore --format=batch-tag < BACKUP
test_expect_equal_file EXPECTED OUTPUT
diff --git a/test/T240-dump-restore.sh b/test/T240-dump-restore.sh
index e6976ff..758d355 100755
--- a/test/T240-dump-restore.sh
+++ b/test/T240-dump-restore.sh
@@ -97,7 +97,7 @@ test_expect_equal_file dump.expected dump.actual
# Note, we assume all messages from cworth have a message-id
# containing cworth.org
-grep 'cworth[.]org' dump.expected > dump-cworth.expected
+(head -1 dump.expected ; grep 'cworth[.]org' dump.expected) > dump-cworth.expected
test_begin_subtest "dump -- from:cworth"
notmuch dump -- from:cworth > dump-dash-cworth.actual
@@ -118,16 +118,16 @@ notmuch search --output=messages from:cworth | sed s/^id:// |\
test_expect_equal_file OUTPUT EXPECTED
test_begin_subtest "format=batch-tag, dump sanity check."
-notmuch dump --format=sup from:cworth | cut -f1 -d' ' | \
+NOTMUCH_DUMP_TAGS --format=sup from:cworth | cut -f1 -d' ' | \
sort > EXPECTED.$test_count
-notmuch dump --format=batch-tag from:cworth | sed 's/^.*-- id://' | \
+NOTMUCH_DUMP_TAGS --format=batch-tag from:cworth | sed 's/^.*-- id://' | \
sort > OUTPUT.$test_count
test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
test_begin_subtest "format=batch-tag, missing newline"
printf "+a_tag_without_newline -- id:20091117232137.GA7669@griffis1.net" > IN
notmuch restore --accumulate < IN
-notmuch dump id:20091117232137.GA7669@griffis1.net > OUT
+NOTMUCH_DUMP_TAGS id:20091117232137.GA7669@griffis1.net > OUT
cat <<EOF > EXPECTED
+a_tag_without_newline +inbox +unread -- id:20091117232137.GA7669@griffis1.net
EOF
@@ -155,7 +155,7 @@ cat <<EOF >EXPECTED.$test_count
+ -- id:20091117232137.GA7669@griffis1.net
EOF
notmuch restore --format=batch-tag < EXPECTED.$test_count
-notmuch dump --format=batch-tag id:20091117232137.GA7669@griffis1.net > OUTPUT.$test_count
+NOTMUCH_DUMP_TAGS --format=batch-tag id:20091117232137.GA7669@griffis1.net > OUTPUT.$test_count
test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
tag1='comic_swear=$&^%$^%\\//-+$^%$'
@@ -217,9 +217,9 @@ notmuch dump --format=batch-tag > OUTPUT.$test_count
test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
test_begin_subtest 'format=batch-tag, checking encoded output'
-notmuch dump --format=batch-tag -- from:cworth |\
+NOTMUCH_DUMP_TAGS --format=batch-tag -- from:cworth |\
awk "{ print \"+$enc1 +$enc2 +$enc3 -- \" \$5 }" > EXPECTED.$test_count
-notmuch dump --format=batch-tag -- from:cworth > OUTPUT.$test_count
+NOTMUCH_DUMP_TAGS --format=batch-tag -- from:cworth > OUTPUT.$test_count
test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
test_begin_subtest 'restoring sane tags'
diff --git a/test/T590-libconfig.sh b/test/T590-libconfig.sh
index 8ca6883..5ea5300 100755
--- a/test/T590-libconfig.sh
+++ b/test/T590-libconfig.sh
@@ -115,4 +115,21 @@ testkey2 testvalue2
EOF
test_expect_equal_file EXPECTED OUTPUT
+test_begin_subtest "dump config"
+cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR}
+{
+ RUN(notmuch_database_set_config (db, "key with spaces", "value, with, spaces!"));
+}
+EOF
+notmuch dump --include=config >OUTPUT
+cat <<'EOF' >EXPECTED
+#notmuch-dump batch-tag:2 config
+#@ aaabefore beforeval
+#@ key%20with%20spaces value,%20with,%20spaces%21
+#@ testkey1 testvalue1
+#@ testkey2 testvalue2
+#@ zzzafter afterval
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
test_done
diff --git a/test/test-lib.sh b/test/test-lib.sh
index 270c718..0c9b366 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -673,6 +673,12 @@ NOTMUCH_NEW ()
notmuch new "${@}" | grep -v -E -e '^Processed [0-9]*( total)? file|Found [0-9]* total file'
}
+NOTMUCH_DUMP_TAGS ()
+{
+ # this relies on the default format being batch-tag, otherwise some tests will break
+ notmuch dump --include=tags "${@}" | sed '/^#/d' | sort
+}
+
notmuch_search_sanitize ()
{
perl -pe 's/("?thread"?: ?)("?)................("?)/\1\2XXX\3/'
--
2.6.4
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 5/5] CLI: optionally restore config data.
2016-01-23 14:59 library config API David Bremner
` (3 preceding siblings ...)
2016-01-23 14:59 ` [PATCH 4/5] CLI: add optional config data to dump output David Bremner
@ 2016-01-23 14:59 ` David Bremner
2016-03-12 12:31 ` library config api v4 David Bremner
5 siblings, 0 replies; 15+ messages in thread
From: David Bremner @ 2016-01-23 14:59 UTC (permalink / raw)
To: notmuch
Like the corresponding change to dump, this needs documentation
---
notmuch-restore.c | 53 ++++++++++++++++++++++++++++++++++++++++++++++++++
test/T590-libconfig.sh | 11 +++++++++++
2 files changed, 64 insertions(+)
diff --git a/notmuch-restore.c b/notmuch-restore.c
index 9abc64f..e06fbde 100644
--- a/notmuch-restore.c
+++ b/notmuch-restore.c
@@ -24,6 +24,38 @@
#include "string-util.h"
#include "zlib-extra.h"
+static int
+process_config_line(notmuch_database_t *notmuch, const char* line){
+ const char *key_p, *val_p;
+ char *key, *val;
+ size_t key_len,val_len;
+ const char *delim=" \t\n";
+ int ret = EXIT_FAILURE;
+
+ void *local = talloc_new(NULL);
+
+ key_p = strtok_len_c(line, delim, &key_len);
+ val_p = strtok_len_c(key_p+key_len, delim, &val_len);
+
+ key = talloc_strndup(local, key_p, key_len);
+ val = talloc_strndup(local, val_p, val_len);
+ if (hex_decode_inplace (key) != HEX_SUCCESS ||
+ hex_decode_inplace (val) != HEX_SUCCESS ) {
+ fprintf (stderr, "hex decoding failure on line %s\n", line);
+ goto DONE;
+ }
+
+ if (print_status_database ("notmuch restore", notmuch,
+ notmuch_database_set_config (notmuch, key, val)))
+ goto DONE;
+
+ ret = EXIT_SUCCESS;
+
+ DONE:
+ talloc_free (local);
+ return ret;
+}
+
static regex_t regex;
/* Non-zero return indicates an error in retrieving the message,
@@ -137,6 +169,7 @@ notmuch_restore_command (notmuch_config_t *config, int argc, char *argv[])
int ret = 0;
int opt_index;
+ int include=0;
int input_format = DUMP_FORMAT_AUTO;
if (notmuch_database_open (notmuch_config_get_database_path (config),
@@ -152,6 +185,10 @@ notmuch_restore_command (notmuch_config_t *config, int argc, char *argv[])
{ "batch-tag", DUMP_FORMAT_BATCH_TAG },
{ "sup", DUMP_FORMAT_SUP },
{ 0, 0 } } },
+ { NOTMUCH_OPT_KEYWORD_FLAGS, &include, "include", 'I',
+ (notmuch_keyword_t []){ { "config", DUMP_INCLUDE_CONFIG },
+ { "tags", DUMP_INCLUDE_TAGS} } },
+
{ NOTMUCH_OPT_STRING, &input_file_name, "input", 'i', 0 },
{ NOTMUCH_OPT_BOOLEAN, &accumulate, "accumulate", 'a', 0 },
{ NOTMUCH_OPT_INHERIT, (void *) ¬much_shared_options, NULL, 0, 0 },
@@ -167,6 +204,10 @@ notmuch_restore_command (notmuch_config_t *config, int argc, char *argv[])
notmuch_process_shared_options (argv[0]);
notmuch_exit_if_unmatched_db_uuid (notmuch);
+ if (include == 0) {
+ include = DUMP_INCLUDE_CONFIG | DUMP_INCLUDE_TAGS;
+ }
+
name_for_error = input_file_name ? input_file_name : "stdin";
if (! accumulate)
@@ -225,11 +266,23 @@ notmuch_restore_command (notmuch_config_t *config, int argc, char *argv[])
ret = EXIT_FAILURE;
goto DONE;
}
+
+ if ((include & DUMP_INCLUDE_CONFIG) && line_len >= 2 && line[0] == '#' && line[1] == '@') {
+ ret = process_config_line(notmuch, line+2);
+ if (ret)
+ goto DONE;
+ }
+
} while ((line_len == 0) ||
(line[0] == '#') ||
/* the cast is safe because we checked about for line_len < 0 */
(strspn (line, " \t\n") == (unsigned)line_len));
+ if (! (include & DUMP_INCLUDE_TAGS)) {
+ ret = EXIT_SUCCESS;
+ goto DONE;
+ }
+
char *p;
for (p = line; (input_format == DUMP_FORMAT_AUTO) && *p; p++) {
if (*p == '(')
diff --git a/test/T590-libconfig.sh b/test/T590-libconfig.sh
index 5ea5300..9c1e566 100755
--- a/test/T590-libconfig.sh
+++ b/test/T590-libconfig.sh
@@ -132,4 +132,15 @@ cat <<'EOF' >EXPECTED
EOF
test_expect_equal_file EXPECTED OUTPUT
+test_begin_subtest "restore config"
+notmuch dump --include=config >EXPECTED
+cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR}
+{
+ RUN(notmuch_database_set_config (db, "testkey1", "mutatedvalue"));
+}
+EOF
+notmuch restore --include=config <EXPECTED
+notmuch dump --include=config >OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
test_done
--
2.6.4
^ permalink raw reply related [flat|nested] 15+ messages in thread
* library config api v4
2016-01-23 14:59 library config API David Bremner
` (4 preceding siblings ...)
2016-01-23 14:59 ` [PATCH 5/5] CLI: optionally restore config data David Bremner
@ 2016-03-12 12:31 ` David Bremner
2016-03-12 12:31 ` [PATCH 1/6] lib: provide config API David Bremner
` (5 more replies)
5 siblings, 6 replies; 15+ messages in thread
From: David Bremner @ 2016-03-12 12:31 UTC (permalink / raw)
To: David Bremner, notmuch
This is an updated version of id:1453561198-2893-1-git-send-email-david@tethera.net
The main change is patch 6/6 that uses the config facility to split the
database from the mail messages. In order to cut the Gordian knot of
how to find the database using config information stored in the
database, this uses a well known location (based on XDG_DATA_HOME) to
store the database. This is a proof of concept at this point; you can
see the basic idea in the last test of T590-libconfig.sh. Some
modifications to notmuch-setup / notmuch-config would be needed to
make this usable.
I haven't tested it very extensively, but in principle at the library
level this allows the database and maildir root to be arbitrary paths;
if a well known location is not used for the database, then it would
be up to the clients to keep track of the database location.
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH 1/6] lib: provide config API
2016-03-12 12:31 ` library config api v4 David Bremner
@ 2016-03-12 12:31 ` David Bremner
2016-03-12 12:31 ` [PATCH 2/6] lib: config list iterators David Bremner
` (4 subsequent siblings)
5 siblings, 0 replies; 15+ messages in thread
From: David Bremner @ 2016-03-12 12:31 UTC (permalink / raw)
To: David Bremner, notmuch
This is a thin wrapper around the Xapian metadata API. The job of this
layer is to keep the config key value pairs from colliding with other
metadata by transparently prefixing the keys, along with the usual glue
to provide a C interface.
The split of _get_config into two functions is to allow returning of the
return value with different memory ownership semantics.
---
lib/Makefile.local | 1 +
lib/config.cc | 90 ++++++++++++++++++++++++++++++++++++++++++++++++++
lib/notmuch.h | 20 +++++++++++
test/T590-libconfig.sh | 58 ++++++++++++++++++++++++++++++++
4 files changed, 169 insertions(+)
create mode 100644 lib/config.cc
create mode 100755 test/T590-libconfig.sh
diff --git a/lib/Makefile.local b/lib/Makefile.local
index 3a07090..eb442d1 100644
--- a/lib/Makefile.local
+++ b/lib/Makefile.local
@@ -48,6 +48,7 @@ libnotmuch_cxx_srcs = \
$(dir)/index.cc \
$(dir)/message.cc \
$(dir)/query.cc \
+ $(dir)/config.cc \
$(dir)/thread.cc
libnotmuch_modules := $(libnotmuch_c_srcs:.c=.o) $(libnotmuch_cxx_srcs:.cc=.o)
diff --git a/lib/config.cc b/lib/config.cc
new file mode 100644
index 0000000..af00d6f
--- /dev/null
+++ b/lib/config.cc
@@ -0,0 +1,90 @@
+/* metadata.cc - API for database metadata
+ *
+ * Copyright © 2015 David Bremner
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see http://www.gnu.org/licenses/ .
+ *
+ * Author: David Bremner <david@tethera.net>
+ */
+
+#include "notmuch.h"
+#include "notmuch-private.h"
+#include "database-private.h"
+
+static const std::string CONFIG_PREFIX="C";
+
+notmuch_status_t
+notmuch_database_set_config (notmuch_database_t *notmuch,
+ const char *key,
+ const char *value)
+{
+ notmuch_status_t status;
+ Xapian::WritableDatabase *db;
+
+ status = _notmuch_database_ensure_writable (notmuch);
+ if (status)
+ return status;
+
+ try {
+ db = static_cast <Xapian::WritableDatabase *> (notmuch->xapian_db);
+ db->set_metadata (CONFIG_PREFIX+key, value);
+ } catch (const Xapian::Error &error) {
+ status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
+ notmuch->exception_reported = TRUE;
+ if (! notmuch->exception_reported) {
+ _notmuch_database_log (notmuch, "Error: A Xapian exception occurred setting metadata: %s\n",
+ error.get_msg().c_str());
+ }
+ }
+ return NOTMUCH_STATUS_SUCCESS;
+}
+
+static notmuch_status_t
+_metadata_value (notmuch_database_t *notmuch,
+ const char *key,
+ std::string &value)
+{
+ notmuch_status_t status = NOTMUCH_STATUS_SUCCESS;
+
+ try {
+ value = notmuch->xapian_db->get_metadata (CONFIG_PREFIX+key);
+ } catch (const Xapian::Error &error) {
+ status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
+ notmuch->exception_reported = TRUE;
+ if (! notmuch->exception_reported) {
+ _notmuch_database_log (notmuch, "Error: A Xapian exception occurred getting metadata: %s\n",
+ error.get_msg().c_str());
+ }
+ }
+ return status;
+}
+
+notmuch_status_t
+notmuch_database_get_config (notmuch_database_t *notmuch,
+ const char *key,
+ char **value) {
+ std::string strval;
+ notmuch_status_t status;
+
+ if (!value)
+ return NOTMUCH_STATUS_NULL_POINTER;
+
+ status = _metadata_value (notmuch, key, strval);
+ if (status)
+ return status;
+
+ *value = strdup (strval.c_str ());
+
+ return NOTMUCH_STATUS_SUCCESS;
+}
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 310a8b8..c62223b 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -1829,6 +1829,26 @@ notmuch_filenames_move_to_next (notmuch_filenames_t *filenames);
void
notmuch_filenames_destroy (notmuch_filenames_t *filenames);
+
+/**
+ * set config 'key' to 'value'
+ *
+ */
+notmuch_status_t
+notmuch_database_set_config (notmuch_database_t *db, const char *key, const char *value);
+
+/**
+ * retrieve config item 'key', assign to 'value'
+ *
+ * keys which have not been previously set with n_d_set_config will
+ * return an empty string.
+ *
+ * return value is allocated by malloc and should be freed by the
+ * caller.
+ */
+notmuch_status_t
+notmuch_database_get_config (notmuch_database_t *db, const char *key, char **value);
+
/* @} */
NOTMUCH_END_DECLS
diff --git a/test/T590-libconfig.sh b/test/T590-libconfig.sh
new file mode 100755
index 0000000..85e4497
--- /dev/null
+++ b/test/T590-libconfig.sh
@@ -0,0 +1,58 @@
+#!/usr/bin/env bash
+test_description="library config API"
+
+. ./test-lib.sh || exit 1
+
+add_email_corpus
+
+cat <<EOF > c_head
+#include <stdio.h>
+#include <string.h>
+#include <stdlib.h>
+#include <notmuch.h>
+
+void run(int line, notmuch_status_t ret)
+{
+ if (ret) {
+ fprintf (stderr, "line %d: %s\n", line, ret);
+ exit (1);
+ }
+}
+
+#define RUN(v) run(__LINE__, v);
+
+int main (int argc, char** argv)
+{
+ notmuch_database_t *db;
+ char *val;
+ notmuch_status_t stat;
+
+ RUN(notmuch_database_open (argv[1], NOTMUCH_DATABASE_MODE_READ_WRITE, &db));
+
+EOF
+
+cat <<EOF > c_tail
+ RUN(notmuch_database_destroy(db));
+}
+EOF
+
+test_begin_subtest "notmuch_database_{set,get}_config"
+cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR}
+{
+ RUN(notmuch_database_set_config (db, "testkey1", "testvalue1"));
+ RUN(notmuch_database_set_config (db, "testkey2", "testvalue2"));
+ RUN(notmuch_database_get_config (db, "testkey1", &val));
+ printf("testkey1 = %s\n", val);
+ RUN(notmuch_database_get_config (db, "testkey2", &val));
+ printf("testkey2 = %s\n", val);
+}
+EOF
+cat <<'EOF' >EXPECTED
+== stdout ==
+testkey1 = testvalue1
+testkey2 = testvalue2
+== stderr ==
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_done
--
2.7.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 2/6] lib: config list iterators
2016-03-12 12:31 ` library config api v4 David Bremner
2016-03-12 12:31 ` [PATCH 1/6] lib: provide config API David Bremner
@ 2016-03-12 12:31 ` David Bremner
2016-03-12 12:31 ` [PATCH 3/6] CLI: add print_status_database David Bremner
` (3 subsequent siblings)
5 siblings, 0 replies; 15+ messages in thread
From: David Bremner @ 2016-03-12 12:31 UTC (permalink / raw)
To: David Bremner, notmuch
Since xapian provides the ability to restrict the iterator to a given
prefix, we expose this ability to the user. Otherwise we mimic the other
iterator interfances in notmuch (e.g. tags.c).
---
lib/config.cc | 104 +++++++++++++++++++++++++++++++++++++++++++++++++
lib/notmuch.h | 44 +++++++++++++++++++++
test/T590-libconfig.sh | 60 ++++++++++++++++++++++++++++
3 files changed, 208 insertions(+)
diff --git a/lib/config.cc b/lib/config.cc
index af00d6f..e581f32 100644
--- a/lib/config.cc
+++ b/lib/config.cc
@@ -24,6 +24,19 @@
static const std::string CONFIG_PREFIX="C";
+struct _notmuch_config_list {
+ notmuch_database_t *notmuch;
+ Xapian::TermIterator *iterator;
+ char *current_key;
+ char *current_val;
+};
+
+static int
+_notmuch_config_list_destroy (notmuch_config_list_t *list) {
+ delete list->iterator;
+ return 0;
+}
+
notmuch_status_t
notmuch_database_set_config (notmuch_database_t *notmuch,
const char *key,
@@ -88,3 +101,94 @@ notmuch_database_get_config (notmuch_database_t *notmuch,
return NOTMUCH_STATUS_SUCCESS;
}
+
+notmuch_status_t
+notmuch_database_get_config_list (notmuch_database_t *notmuch,
+ const char *prefix,
+ notmuch_config_list_t **out)
+{
+ notmuch_config_list_t *list = NULL;
+ notmuch_status_t status = NOTMUCH_STATUS_SUCCESS;
+
+ list = talloc (notmuch, notmuch_config_list_t);
+ if (!list) {
+ status = NOTMUCH_STATUS_OUT_OF_MEMORY;
+ goto DONE;
+ }
+
+ talloc_set_destructor (list, _notmuch_config_list_destroy);
+ list->iterator = new Xapian::TermIterator;
+ list->notmuch = notmuch;
+ list->current_key = NULL;
+ list->current_val = NULL;
+
+ try {
+
+ *list->iterator = notmuch->xapian_db->metadata_keys_begin (CONFIG_PREFIX + (prefix ? prefix : ""));
+
+ } catch (const Xapian::Error &error) {
+ _notmuch_database_log (notmuch, "A Xapian exception occurred getting metadata iterator: %s.\n",
+ error.get_msg().c_str());
+ notmuch->exception_reported = TRUE;
+ status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
+ }
+
+ *out = list;
+
+ DONE:
+ if (status && list)
+ talloc_free (list);
+
+ return status;
+}
+
+notmuch_bool_t
+notmuch_config_list_valid (notmuch_config_list_t *metadata)
+{
+ if (*(metadata->iterator) == metadata->notmuch->xapian_db->metadata_keys_end())
+ return FALSE;
+
+ return TRUE;
+}
+
+const char *
+notmuch_config_list_key (notmuch_config_list_t *list)
+{
+ if (list->current_key)
+ talloc_free (list->current_key);
+
+ list->current_key = talloc_strdup (list, (**(list->iterator)).c_str () + CONFIG_PREFIX.length ());
+
+ return list->current_key;
+}
+
+const char *
+notmuch_config_list_value (notmuch_config_list_t *list)
+{
+ std::string strval;
+ notmuch_status_t status;
+ const char *key = notmuch_config_list_key (list);
+
+ /* TODO: better error reporting?? */
+ status = _metadata_value (list->notmuch, key, strval);
+ if (status)
+ return NULL;
+
+ if (list->current_val)
+ talloc_free(list->current_val);
+
+ list->current_val = talloc_strdup(list, strval.c_str ());
+ return list->current_val;
+}
+
+void
+notmuch_config_list_move_to_next (notmuch_config_list_t *list)
+{
+ (*(list->iterator))++;
+}
+
+void
+notmuch_config_list_destroy (notmuch_config_list_t *list)
+{
+ talloc_free (list);
+}
diff --git a/lib/notmuch.h b/lib/notmuch.h
index c62223b..b439c88 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -197,6 +197,7 @@ typedef struct _notmuch_message notmuch_message_t;
typedef struct _notmuch_tags notmuch_tags_t;
typedef struct _notmuch_directory notmuch_directory_t;
typedef struct _notmuch_filenames notmuch_filenames_t;
+typedef struct _notmuch_config_list notmuch_config_list_t;
#endif /* __DOXYGEN__ */
/**
@@ -1849,6 +1850,49 @@ notmuch_database_set_config (notmuch_database_t *db, const char *key, const char
notmuch_status_t
notmuch_database_get_config (notmuch_database_t *db, const char *key, char **value);
+/**
+ * Create an iterator for all config items with keys matching a given prefix
+ */
+notmuch_status_t
+notmuch_database_get_config_list (notmuch_database_t *db, const char *prefix, notmuch_config_list_t **out);
+
+/**
+ * Is 'config_list' iterator valid (i.e. _key, _value, _move_to_next can be called).
+ */
+notmuch_bool_t
+notmuch_config_list_valid (notmuch_config_list_t *config_list);
+
+/**
+ * return key for current config pair
+ *
+ * return value is owned by the iterator, and will be destroyed by the
+ * next call to notmuch_config_list_key or notmuch_config_list_destroy.
+ */
+const char *
+notmuch_config_list_key (notmuch_config_list_t *config_list);
+
+/**
+ * return 'value' for current config pair
+ *
+ * return value is owned by the iterator, and will be destroyed by the
+ * next call to notmuch_config_list_value or notmuch config_list_destroy
+ */
+const char *
+notmuch_config_list_value (notmuch_config_list_t *config_list);
+
+
+/**
+ * move 'config_list' iterator to the next pair
+ */
+void
+notmuch_config_list_move_to_next (notmuch_config_list_t *config_list);
+
+/**
+ * free any resources held by 'config_list'
+ */
+void
+notmuch_config_list_destroy (notmuch_config_list_t *config_list);
+
/* @} */
NOTMUCH_END_DECLS
diff --git a/test/T590-libconfig.sh b/test/T590-libconfig.sh
index 85e4497..8ca6883 100755
--- a/test/T590-libconfig.sh
+++ b/test/T590-libconfig.sh
@@ -55,4 +55,64 @@ testkey2 = testvalue2
EOF
test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "notmuch_database_get_config_list: empty list"
+cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR}
+{
+ notmuch_config_list_t *list;
+ RUN(notmuch_database_get_config_list (db, "nonexistent", &list));
+ printf("valid = %d\n", notmuch_config_list_valid (list));
+ notmuch_config_list_destroy (list);
+}
+EOF
+cat <<'EOF' >EXPECTED
+== stdout ==
+valid = 0
+== stderr ==
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+
+test_begin_subtest "notmuch_database_get_config_list: all pairs"
+cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR}
+{
+ notmuch_config_list_t *list;
+ RUN(notmuch_database_set_config (db, "zzzafter", "afterval"));
+ RUN(notmuch_database_set_config (db, "aaabefore", "beforeval"));
+ RUN(notmuch_database_get_config_list (db, "", &list));
+ for (; notmuch_config_list_valid (list); notmuch_config_list_move_to_next (list)) {
+ printf("%s %s\n", notmuch_config_list_key (list), notmuch_config_list_value(list));
+ }
+ notmuch_config_list_destroy (list);
+}
+EOF
+cat <<'EOF' >EXPECTED
+== stdout ==
+aaabefore beforeval
+testkey1 testvalue1
+testkey2 testvalue2
+zzzafter afterval
+== stderr ==
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "notmuch_database_get_config_list: one prefix"
+cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR}
+{
+ notmuch_config_list_t *list;
+ RUN(notmuch_database_get_config_list (db, "testkey", &list));
+ for (; notmuch_config_list_valid (list); notmuch_config_list_move_to_next (list)) {
+ printf("%s %s\n", notmuch_config_list_key (list), notmuch_config_list_value(list));
+ }
+ notmuch_config_list_destroy (list);
+}
+EOF
+cat <<'EOF' >EXPECTED
+== stdout ==
+testkey1 testvalue1
+testkey2 testvalue2
+== stderr ==
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
test_done
--
2.7.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 3/6] CLI: add print_status_database
2016-03-12 12:31 ` library config api v4 David Bremner
2016-03-12 12:31 ` [PATCH 1/6] lib: provide config API David Bremner
2016-03-12 12:31 ` [PATCH 2/6] lib: config list iterators David Bremner
@ 2016-03-12 12:31 ` David Bremner
2016-03-12 12:31 ` [PATCH 4/6] CLI: add optional config data to dump output David Bremner
` (2 subsequent siblings)
5 siblings, 0 replies; 15+ messages in thread
From: David Bremner @ 2016-03-12 12:31 UTC (permalink / raw)
To: David Bremner, notmuch
This could probably be used at quite a few places in the existing code,
but in the immediate future I plan to use in some new code in
notmuch-dump
---
notmuch-client.h | 5 +++++
status.c | 17 +++++++++++++++++
2 files changed, 22 insertions(+)
diff --git a/notmuch-client.h b/notmuch-client.h
index 18e6c60..b3d0b66 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -462,6 +462,11 @@ print_status_query (const char *loc,
const notmuch_query_t *query,
notmuch_status_t status);
+notmuch_status_t
+print_status_database (const char *loc,
+ const notmuch_database_t *database,
+ notmuch_status_t status);
+
#include "command-line-arguments.h"
extern char *notmuch_requested_db_uuid;
diff --git a/status.c b/status.c
index 8fa81cb..45d3fb4 100644
--- a/status.c
+++ b/status.c
@@ -19,3 +19,20 @@ print_status_query (const char *loc,
}
return status;
}
+
+notmuch_status_t
+print_status_database (const char *loc,
+ const notmuch_database_t *notmuch,
+ notmuch_status_t status)
+{
+ if (status) {
+ const char *msg;
+
+ fprintf (stderr, "%s: %s\n", loc,
+ notmuch_status_to_string (status));
+ msg = notmuch_database_status_string (notmuch);
+ if (msg)
+ fputs (msg, stderr);
+ }
+ return status;
+}
--
2.7.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 4/6] CLI: add optional config data to dump output.
2016-03-12 12:31 ` library config api v4 David Bremner
` (2 preceding siblings ...)
2016-03-12 12:31 ` [PATCH 3/6] CLI: add print_status_database David Bremner
@ 2016-03-12 12:31 ` David Bremner
2016-03-12 12:31 ` [PATCH 5/6] CLI: optionally restore config data David Bremner
2016-03-12 12:31 ` [PATCH 6/6] WIP: support XDG database directory David Bremner
5 siblings, 0 replies; 15+ messages in thread
From: David Bremner @ 2016-03-12 12:31 UTC (permalink / raw)
To: David Bremner, notmuch
This lacks at least documentation. Note that it changes the default dump
output format, but doesn't break existing notmuch-restore. It might
break user scripts though.
---
notmuch-client.h | 8 +++++
notmuch-dump.c | 80 +++++++++++++++++++++++++++++++++++++++++++++--
notmuch-new.c | 2 +-
test/T150-tagging.sh | 8 ++---
test/T240-dump-restore.sh | 14 ++++-----
test/T590-libconfig.sh | 17 ++++++++++
test/test-lib.sh | 6 ++++
7 files changed, 120 insertions(+), 15 deletions(-)
diff --git a/notmuch-client.h b/notmuch-client.h
index b3d0b66..ae6f124 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -446,11 +446,19 @@ typedef enum dump_formats {
DUMP_FORMAT_SUP
} dump_format_t;
+typedef enum dump_includes {
+ DUMP_INCLUDE_TAGS=1,
+ DUMP_INCLUDE_CONFIG=2,
+} dump_include_t;
+
+#define NOTMUCH_DUMP_VERSION 2
+
int
notmuch_database_dump (notmuch_database_t *notmuch,
const char *output_file_name,
const char *query_str,
dump_format_t output_format,
+ dump_include_t include,
notmuch_bool_t gzip_output);
/* If status is non-zero (i.e. error) print appropriate
diff --git a/notmuch-dump.c b/notmuch-dump.c
index 829781f..a6cf810 100644
--- a/notmuch-dump.c
+++ b/notmuch-dump.c
@@ -23,16 +23,82 @@
#include "string-util.h"
#include <zlib.h>
+static int
+database_dump_config (notmuch_database_t *notmuch, gzFile output)
+{
+ notmuch_config_list_t *list;
+ int ret = EXIT_FAILURE;
+ char *buffer = NULL;
+ size_t buffer_size = 0;
+
+ if (print_status_database ("notmuch dump", notmuch,
+ notmuch_database_get_config_list (notmuch, NULL, &list)))
+ goto DONE;
+
+ for (; notmuch_config_list_valid (list); notmuch_config_list_move_to_next (list)) {
+ if (hex_encode (notmuch, notmuch_config_list_key (list),
+ &buffer, &buffer_size) != HEX_SUCCESS) {
+ fprintf (stderr, "Error: failed to hex-encode config key %s\n",
+ notmuch_config_list_key (list));
+ goto DONE;
+ }
+ gzprintf (output, "#@ %s", buffer);
+
+ if (hex_encode (notmuch, notmuch_config_list_value (list),
+ &buffer, &buffer_size) != HEX_SUCCESS) {
+ fprintf (stderr, "Error: failed to hex-encode config value %s\n",
+ notmuch_config_list_value (list) );
+ goto DONE;
+ }
+
+ gzprintf (output, " %s\n", buffer);
+ }
+
+ ret = EXIT_SUCCESS;
+
+ DONE:
+ if (list)
+ notmuch_config_list_destroy (list);
+
+ if (buffer)
+ talloc_free (buffer);
+
+ return ret;
+}
+
+static void
+print_dump_header (gzFile output, int output_format, int include)
+{
+ gzprintf (output, "#notmuch-dump %s:%d %s%s%s\n",
+ (output_format == DUMP_FORMAT_SUP) ? "sup" : "batch-tag",
+ NOTMUCH_DUMP_VERSION,
+ (include & DUMP_INCLUDE_CONFIG) ? "config" : "",
+ (include & DUMP_INCLUDE_TAGS) && (include & DUMP_INCLUDE_CONFIG) ? "," : "",
+ (include & DUMP_INCLUDE_TAGS) ? "tags" : "");
+
+
+}
static int
database_dump_file (notmuch_database_t *notmuch, gzFile output,
- const char *query_str, int output_format)
+ const char *query_str, int output_format, int include)
{
notmuch_query_t *query;
notmuch_messages_t *messages;
notmuch_message_t *message;
notmuch_tags_t *tags;
+ print_dump_header (output, output_format, include);
+
+ if (include & DUMP_INCLUDE_CONFIG) {
+ if (print_status_database ("notmuch dump", notmuch,
+ database_dump_config(notmuch,output)))
+ return EXIT_FAILURE;
+ }
+
+ if (! (include & DUMP_INCLUDE_TAGS))
+ return EXIT_SUCCESS;
+
if (! query_str)
query_str = "";
@@ -130,6 +196,7 @@ notmuch_database_dump (notmuch_database_t *notmuch,
const char *output_file_name,
const char *query_str,
dump_format_t output_format,
+ dump_include_t include,
notmuch_bool_t gzip_output)
{
gzFile output = NULL;
@@ -164,7 +231,7 @@ notmuch_database_dump (notmuch_database_t *notmuch,
goto DONE;
}
- ret = database_dump_file (notmuch, output, query_str, output_format);
+ ret = database_dump_file (notmuch, output, query_str, output_format, include);
if (ret) goto DONE;
ret = gzflush (output, Z_FINISH);
@@ -226,6 +293,7 @@ notmuch_dump_command (notmuch_config_t *config, int argc, char *argv[])
int opt_index;
int output_format = DUMP_FORMAT_BATCH_TAG;
+ int include = 0;
notmuch_bool_t gzip_output = 0;
notmuch_opt_desc_t options[] = {
@@ -233,6 +301,9 @@ notmuch_dump_command (notmuch_config_t *config, int argc, char *argv[])
(notmuch_keyword_t []){ { "sup", DUMP_FORMAT_SUP },
{ "batch-tag", DUMP_FORMAT_BATCH_TAG },
{ 0, 0 } } },
+ { NOTMUCH_OPT_KEYWORD_FLAGS, &include, "include", 'I',
+ (notmuch_keyword_t []){ { "config", DUMP_INCLUDE_CONFIG },
+ { "tags", DUMP_INCLUDE_TAGS} } },
{ NOTMUCH_OPT_STRING, &output_file_name, "output", 'o', 0 },
{ NOTMUCH_OPT_BOOLEAN, &gzip_output, "gzip", 'z', 0 },
{ NOTMUCH_OPT_INHERIT, (void *) ¬much_shared_options, NULL, 0, 0 },
@@ -245,6 +316,9 @@ notmuch_dump_command (notmuch_config_t *config, int argc, char *argv[])
notmuch_process_shared_options (argv[0]);
+ if (include == 0)
+ include = DUMP_INCLUDE_CONFIG | DUMP_INCLUDE_TAGS;
+
if (opt_index < argc) {
query_str = query_string_from_args (notmuch, argc - opt_index, argv + opt_index);
if (query_str == NULL) {
@@ -254,7 +328,7 @@ notmuch_dump_command (notmuch_config_t *config, int argc, char *argv[])
}
ret = notmuch_database_dump (notmuch, output_file_name, query_str,
- output_format, gzip_output);
+ output_format, include, gzip_output);
notmuch_database_destroy (notmuch);
diff --git a/notmuch-new.c b/notmuch-new.c
index e503776..fd2ff82 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -1041,7 +1041,7 @@ notmuch_new_command (notmuch_config_t *config, int argc, char *argv[])
}
if (notmuch_database_dump (notmuch, backup_name, "",
- DUMP_FORMAT_BATCH_TAG, TRUE)) {
+ DUMP_FORMAT_BATCH_TAG, DUMP_INCLUDE_CONFIG | DUMP_INCLUDE_TAGS, TRUE)) {
fprintf (stderr, "Backup failed. Aborting upgrade.");
return EXIT_FAILURE;
}
diff --git a/test/T150-tagging.sh b/test/T150-tagging.sh
index 8adcabc..f1a6df0 100755
--- a/test/T150-tagging.sh
+++ b/test/T150-tagging.sh
@@ -188,7 +188,7 @@ cat <<EOF > EXPECTED
+%22%27%22%27%22%22%27%27 +inbox +tag4 +tag5 +unread -- id:msg-002@notmuch-test-suite
EOF
-notmuch dump --format=batch-tag | sort > OUTPUT
+NOTMUCH_DUMP_TAGS > OUTPUT
notmuch restore --format=batch-tag < BACKUP
test_expect_equal_file EXPECTED OUTPUT
@@ -209,7 +209,7 @@ cat <<EOF > EXPECTED
+%21@%23%20%24%25%5e%26%2a%29-_=+%5b%7b%5c%20%7c%3b%3a%27%20%22,.%3c%60%7e +inbox +tag5 +unread -- id:msg-001@notmuch-test-suite
EOF
-notmuch dump --format=batch-tag | sort > OUTPUT
+NOTMUCH_DUMP_TAGS > OUTPUT
notmuch restore --format=batch-tag < BACKUP
test_expect_equal_file EXPECTED OUTPUT
@@ -235,7 +235,7 @@ cat <<EOF > EXPECTED
+%2a@%7d%cf%b5%f4%85%80%adO3%da%a7 +=%e0%ac%95%c8%b3+%ef%aa%95%c8%a64w%c7%9d%c9%a2%cf%b3%d6%82%24B%c4%a9%c5%a1UX%ee%99%b0%27E7%ca%a4%d0%8b%5d +A%e1%a0%bc%de%8b%d5%b2V%d9%9b%f3%b5%a2%a3M%d8%a1u@%f0%a0%ac%948%7e%f0%ab%86%af%27 +L%df%85%ef%a1%a5m@%d3%96%c2%ab%d4%9f%ca%b8%f3%b3%a2%bf%c7%b1_u%d7%b4%c7%b1 +P%c4%98%2f +R +inbox +tag5 +unread +%7e%d1%8b%25%ec%a0%ae%d1%a0M%3b%e3%b6%b7%e9%a4%87%3c%db%9a%cc%a8%e1%96%9d +%c4%bf7%c7%ab9H%c4%99k%ea%91%bd%c3%8ck%e2%b3%8dk%c5%952V%e4%99%b2%d9%b3%e4%8b%bda%5b%24%c7%9b +%da%88=f%cc%b9I%ce%af%7b%c9%97%e3%b9%8bH%cb%92X%d2%8c6 +%dc%9crh%d2%86B%e5%97%a2%22t%ed%99%82d -- id:msg-001@notmuch-test-suite
EOF
-notmuch dump --format=batch-tag | sort > OUTPUT
+NOTMUCH_DUMP_TAGS > OUTPUT
notmuch restore --format=batch-tag < BACKUP
test_expect_equal_file EXPECTED OUTPUT
@@ -260,7 +260,7 @@ cat <<EOF > EXPECTED
+foo%3a%3abar%25 +found%3a%3ait +inbox +tag5 +unread +winner -- id:msg-001@notmuch-test-suite
EOF
-notmuch dump --format=batch-tag | sort > OUTPUT
+NOTMUCH_DUMP_TAGS > OUTPUT
notmuch restore --format=batch-tag < BACKUP
test_expect_equal_file EXPECTED OUTPUT
diff --git a/test/T240-dump-restore.sh b/test/T240-dump-restore.sh
index e6976ff..758d355 100755
--- a/test/T240-dump-restore.sh
+++ b/test/T240-dump-restore.sh
@@ -97,7 +97,7 @@ test_expect_equal_file dump.expected dump.actual
# Note, we assume all messages from cworth have a message-id
# containing cworth.org
-grep 'cworth[.]org' dump.expected > dump-cworth.expected
+(head -1 dump.expected ; grep 'cworth[.]org' dump.expected) > dump-cworth.expected
test_begin_subtest "dump -- from:cworth"
notmuch dump -- from:cworth > dump-dash-cworth.actual
@@ -118,16 +118,16 @@ notmuch search --output=messages from:cworth | sed s/^id:// |\
test_expect_equal_file OUTPUT EXPECTED
test_begin_subtest "format=batch-tag, dump sanity check."
-notmuch dump --format=sup from:cworth | cut -f1 -d' ' | \
+NOTMUCH_DUMP_TAGS --format=sup from:cworth | cut -f1 -d' ' | \
sort > EXPECTED.$test_count
-notmuch dump --format=batch-tag from:cworth | sed 's/^.*-- id://' | \
+NOTMUCH_DUMP_TAGS --format=batch-tag from:cworth | sed 's/^.*-- id://' | \
sort > OUTPUT.$test_count
test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
test_begin_subtest "format=batch-tag, missing newline"
printf "+a_tag_without_newline -- id:20091117232137.GA7669@griffis1.net" > IN
notmuch restore --accumulate < IN
-notmuch dump id:20091117232137.GA7669@griffis1.net > OUT
+NOTMUCH_DUMP_TAGS id:20091117232137.GA7669@griffis1.net > OUT
cat <<EOF > EXPECTED
+a_tag_without_newline +inbox +unread -- id:20091117232137.GA7669@griffis1.net
EOF
@@ -155,7 +155,7 @@ cat <<EOF >EXPECTED.$test_count
+ -- id:20091117232137.GA7669@griffis1.net
EOF
notmuch restore --format=batch-tag < EXPECTED.$test_count
-notmuch dump --format=batch-tag id:20091117232137.GA7669@griffis1.net > OUTPUT.$test_count
+NOTMUCH_DUMP_TAGS --format=batch-tag id:20091117232137.GA7669@griffis1.net > OUTPUT.$test_count
test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
tag1='comic_swear=$&^%$^%\\//-+$^%$'
@@ -217,9 +217,9 @@ notmuch dump --format=batch-tag > OUTPUT.$test_count
test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
test_begin_subtest 'format=batch-tag, checking encoded output'
-notmuch dump --format=batch-tag -- from:cworth |\
+NOTMUCH_DUMP_TAGS --format=batch-tag -- from:cworth |\
awk "{ print \"+$enc1 +$enc2 +$enc3 -- \" \$5 }" > EXPECTED.$test_count
-notmuch dump --format=batch-tag -- from:cworth > OUTPUT.$test_count
+NOTMUCH_DUMP_TAGS --format=batch-tag -- from:cworth > OUTPUT.$test_count
test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
test_begin_subtest 'restoring sane tags'
diff --git a/test/T590-libconfig.sh b/test/T590-libconfig.sh
index 8ca6883..5ea5300 100755
--- a/test/T590-libconfig.sh
+++ b/test/T590-libconfig.sh
@@ -115,4 +115,21 @@ testkey2 testvalue2
EOF
test_expect_equal_file EXPECTED OUTPUT
+test_begin_subtest "dump config"
+cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR}
+{
+ RUN(notmuch_database_set_config (db, "key with spaces", "value, with, spaces!"));
+}
+EOF
+notmuch dump --include=config >OUTPUT
+cat <<'EOF' >EXPECTED
+#notmuch-dump batch-tag:2 config
+#@ aaabefore beforeval
+#@ key%20with%20spaces value,%20with,%20spaces%21
+#@ testkey1 testvalue1
+#@ testkey2 testvalue2
+#@ zzzafter afterval
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
test_done
diff --git a/test/test-lib.sh b/test/test-lib.sh
index cc08a98..61bad0d 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -673,6 +673,12 @@ NOTMUCH_NEW ()
notmuch new "${@}" | grep -v -E -e '^Processed [0-9]*( total)? file|Found [0-9]* total file'
}
+NOTMUCH_DUMP_TAGS ()
+{
+ # this relies on the default format being batch-tag, otherwise some tests will break
+ notmuch dump --include=tags "${@}" | sed '/^#/d' | sort
+}
+
notmuch_search_sanitize ()
{
perl -pe 's/("?thread"?: ?)("?)................("?)/\1\2XXX\3/'
--
2.7.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 5/6] CLI: optionally restore config data.
2016-03-12 12:31 ` library config api v4 David Bremner
` (3 preceding siblings ...)
2016-03-12 12:31 ` [PATCH 4/6] CLI: add optional config data to dump output David Bremner
@ 2016-03-12 12:31 ` David Bremner
2016-03-12 12:31 ` [PATCH 6/6] WIP: support XDG database directory David Bremner
5 siblings, 0 replies; 15+ messages in thread
From: David Bremner @ 2016-03-12 12:31 UTC (permalink / raw)
To: David Bremner, notmuch
Like the corresponding change to dump, this needs documentation
---
notmuch-restore.c | 53 ++++++++++++++++++++++++++++++++++++++++++++++++++
test/T590-libconfig.sh | 11 +++++++++++
2 files changed, 64 insertions(+)
diff --git a/notmuch-restore.c b/notmuch-restore.c
index 9abc64f..e06fbde 100644
--- a/notmuch-restore.c
+++ b/notmuch-restore.c
@@ -24,6 +24,38 @@
#include "string-util.h"
#include "zlib-extra.h"
+static int
+process_config_line(notmuch_database_t *notmuch, const char* line){
+ const char *key_p, *val_p;
+ char *key, *val;
+ size_t key_len,val_len;
+ const char *delim=" \t\n";
+ int ret = EXIT_FAILURE;
+
+ void *local = talloc_new(NULL);
+
+ key_p = strtok_len_c(line, delim, &key_len);
+ val_p = strtok_len_c(key_p+key_len, delim, &val_len);
+
+ key = talloc_strndup(local, key_p, key_len);
+ val = talloc_strndup(local, val_p, val_len);
+ if (hex_decode_inplace (key) != HEX_SUCCESS ||
+ hex_decode_inplace (val) != HEX_SUCCESS ) {
+ fprintf (stderr, "hex decoding failure on line %s\n", line);
+ goto DONE;
+ }
+
+ if (print_status_database ("notmuch restore", notmuch,
+ notmuch_database_set_config (notmuch, key, val)))
+ goto DONE;
+
+ ret = EXIT_SUCCESS;
+
+ DONE:
+ talloc_free (local);
+ return ret;
+}
+
static regex_t regex;
/* Non-zero return indicates an error in retrieving the message,
@@ -137,6 +169,7 @@ notmuch_restore_command (notmuch_config_t *config, int argc, char *argv[])
int ret = 0;
int opt_index;
+ int include=0;
int input_format = DUMP_FORMAT_AUTO;
if (notmuch_database_open (notmuch_config_get_database_path (config),
@@ -152,6 +185,10 @@ notmuch_restore_command (notmuch_config_t *config, int argc, char *argv[])
{ "batch-tag", DUMP_FORMAT_BATCH_TAG },
{ "sup", DUMP_FORMAT_SUP },
{ 0, 0 } } },
+ { NOTMUCH_OPT_KEYWORD_FLAGS, &include, "include", 'I',
+ (notmuch_keyword_t []){ { "config", DUMP_INCLUDE_CONFIG },
+ { "tags", DUMP_INCLUDE_TAGS} } },
+
{ NOTMUCH_OPT_STRING, &input_file_name, "input", 'i', 0 },
{ NOTMUCH_OPT_BOOLEAN, &accumulate, "accumulate", 'a', 0 },
{ NOTMUCH_OPT_INHERIT, (void *) ¬much_shared_options, NULL, 0, 0 },
@@ -167,6 +204,10 @@ notmuch_restore_command (notmuch_config_t *config, int argc, char *argv[])
notmuch_process_shared_options (argv[0]);
notmuch_exit_if_unmatched_db_uuid (notmuch);
+ if (include == 0) {
+ include = DUMP_INCLUDE_CONFIG | DUMP_INCLUDE_TAGS;
+ }
+
name_for_error = input_file_name ? input_file_name : "stdin";
if (! accumulate)
@@ -225,11 +266,23 @@ notmuch_restore_command (notmuch_config_t *config, int argc, char *argv[])
ret = EXIT_FAILURE;
goto DONE;
}
+
+ if ((include & DUMP_INCLUDE_CONFIG) && line_len >= 2 && line[0] == '#' && line[1] == '@') {
+ ret = process_config_line(notmuch, line+2);
+ if (ret)
+ goto DONE;
+ }
+
} while ((line_len == 0) ||
(line[0] == '#') ||
/* the cast is safe because we checked about for line_len < 0 */
(strspn (line, " \t\n") == (unsigned)line_len));
+ if (! (include & DUMP_INCLUDE_TAGS)) {
+ ret = EXIT_SUCCESS;
+ goto DONE;
+ }
+
char *p;
for (p = line; (input_format == DUMP_FORMAT_AUTO) && *p; p++) {
if (*p == '(')
diff --git a/test/T590-libconfig.sh b/test/T590-libconfig.sh
index 5ea5300..9c1e566 100755
--- a/test/T590-libconfig.sh
+++ b/test/T590-libconfig.sh
@@ -132,4 +132,15 @@ cat <<'EOF' >EXPECTED
EOF
test_expect_equal_file EXPECTED OUTPUT
+test_begin_subtest "restore config"
+notmuch dump --include=config >EXPECTED
+cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR}
+{
+ RUN(notmuch_database_set_config (db, "testkey1", "mutatedvalue"));
+}
+EOF
+notmuch restore --include=config <EXPECTED
+notmuch dump --include=config >OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
test_done
--
2.7.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 6/6] WIP: support XDG database directory
2016-03-12 12:31 ` library config api v4 David Bremner
` (4 preceding siblings ...)
2016-03-12 12:31 ` [PATCH 5/6] CLI: optionally restore config data David Bremner
@ 2016-03-12 12:31 ` David Bremner
2016-03-14 18:06 ` Tomi Ollila
5 siblings, 1 reply; 15+ messages in thread
From: David Bremner @ 2016-03-12 12:31 UTC (permalink / raw)
To: David Bremner, notmuch
---
lib/database.cc | 66 +++++++++++++++++++++++++++++++++++++++++---------
test/T560-lib-error.sh | 2 +-
test/T590-libconfig.sh | 35 ++++++++++++++++++++++++++
3 files changed, 90 insertions(+), 13 deletions(-)
diff --git a/lib/database.cc b/lib/database.cc
index 3b342f1..3d19bec 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -855,6 +855,23 @@ notmuch_database_open (const char *path,
return status;
}
+static char *
+_xdg_database_path (void *ctx) {
+
+ const char *data_dir = NULL;
+
+ data_dir = getenv ("XDG_DATA_HOME");
+
+ if (! data_dir) {
+ const char *home = getenv ("HOME");
+ if (! home)
+ return NULL;
+
+ data_dir = talloc_asprintf (ctx, "%s/.local/share", home);
+ }
+ return talloc_asprintf (ctx, "%s/notmuch", data_dir);
+}
+
notmuch_status_t
notmuch_database_open_verbose (const char *path,
notmuch_database_mode_t mode,
@@ -865,6 +882,7 @@ notmuch_database_open_verbose (const char *path,
void *local = talloc_new (NULL);
notmuch_database_t *notmuch = NULL;
char *notmuch_path, *xapian_path, *incompat_features;
+ char *xdg_path = NULL;
char *message = NULL;
struct stat st;
int err;
@@ -872,21 +890,29 @@ notmuch_database_open_verbose (const char *path,
static int initialized = 0;
if (path == NULL) {
- message = strdup ("Error: Cannot open a database for a NULL path.\n");
- status = NOTMUCH_STATUS_NULL_POINTER;
- goto DONE;
+ xdg_path = _xdg_database_path (local);
+ if (! xdg_path) {
+ message = strdup ("Error: NULL path, and cannot compute XDG_DATA_HOME.\n");
+ status = NOTMUCH_STATUS_NULL_POINTER;
+ goto DONE;
+ }
}
- if (path[0] != '/') {
+ if (path && path[0] != '/') {
message = strdup ("Error: Database path must be absolute.\n");
status = NOTMUCH_STATUS_PATH_ERROR;
goto DONE;
}
- if (! (notmuch_path = talloc_asprintf (local, "%s/%s", path, ".notmuch"))) {
- message = strdup ("Out of memory\n");
- status = NOTMUCH_STATUS_OUT_OF_MEMORY;
- goto DONE;
+ if (xdg_path) {
+ notmuch_path = xdg_path;
+ } else {
+ notmuch_path = talloc_asprintf (local, "%s/%s", path, ".notmuch");
+ if (! (notmuch_path)) {
+ message = strdup ("Out of memory\n");
+ status = NOTMUCH_STATUS_OUT_OF_MEMORY;
+ goto DONE;
+ }
}
err = stat (notmuch_path, &st);
@@ -917,10 +943,14 @@ notmuch_database_open_verbose (const char *path,
notmuch = talloc_zero (NULL, notmuch_database_t);
notmuch->exception_reported = FALSE;
notmuch->status_string = NULL;
- notmuch->path = talloc_strdup (notmuch, path);
+ if (path) {
+ notmuch->path = talloc_strdup (notmuch, path);
- if (notmuch->path[strlen (notmuch->path) - 1] == '/')
- notmuch->path[strlen (notmuch->path) - 1] = '\0';
+ if (notmuch->path[strlen (notmuch->path) - 1] == '/')
+ notmuch->path[strlen (notmuch->path) - 1] = '\0';
+ } else {
+ notmuch->path = NULL;
+ }
notmuch->mode = mode;
notmuch->atomic_nesting = 0;
@@ -1303,7 +1333,19 @@ notmuch_database_destroy (notmuch_database_t *notmuch)
const char *
notmuch_database_get_path (notmuch_database_t *notmuch)
{
- return notmuch->path;
+ char *path = NULL;
+ notmuch_status_t status;
+
+ if (notmuch->path)
+ return notmuch->path;
+
+ status = notmuch_database_get_config (notmuch, "maildir_root", &path);
+ if (status) {
+ _notmuch_database_log (notmuch, "unable to find maildir_root\n");
+ return NULL;
+ }
+
+ return path;
}
unsigned int
diff --git a/test/T560-lib-error.sh b/test/T560-lib-error.sh
index 59a479c..8d4eaf5 100755
--- a/test/T560-lib-error.sh
+++ b/test/T560-lib-error.sh
@@ -21,7 +21,7 @@ EOF
cat <<'EOF' >EXPECTED
== stdout ==
== stderr ==
-Error: Cannot open a database for a NULL path.
+Error opening database at CWD/home/.local/share/notmuch: No such file or directory
EOF
test_expect_equal_file EXPECTED OUTPUT
diff --git a/test/T590-libconfig.sh b/test/T590-libconfig.sh
index 9c1e566..52b06bb 100755
--- a/test/T590-libconfig.sh
+++ b/test/T590-libconfig.sh
@@ -143,4 +143,39 @@ notmuch restore --include=config <EXPECTED
notmuch dump --include=config >OUTPUT
test_expect_equal_file EXPECTED OUTPUT
+XDG_DIR=$HOME/.local/share/notmuch
+test_begin_subtest "Split database and maildir"
+xapian-metadata set ${MAIL_DIR}/.notmuch/xapian Cmaildir_root ${MAIL_DIR}
+mkdir -p $XDG_DIR
+mv ${MAIL_DIR}/.notmuch/xapian $XDG_DIR
+test_C <<EOF >OUTPUT
+#include <stdio.h>
+#include <notmuch.h>
+
+int main (int argc, char** argv)
+{
+ notmuch_database_t *db;
+ char *val;
+ notmuch_status_t stat;
+ notmuch_message_t *message;
+
+ stat=notmuch_database_open (NULL, NOTMUCH_DATABASE_MODE_READ_WRITE, &db);
+ printf("database_open status = %d\n", stat);
+ stat = notmuch_database_find_message (db, "87ocn0qh6d.fsf@yoom.home.cworth.org", &message);
+ printf("find_message status = %d\n", stat);
+ printf("found message = %d\n", message != NULL);
+ printf("filename = %s\n",notmuch_message_get_filename (message));
+}
+EOF
+
+cat <<EOF >EXPECTED
+== stdout ==
+database_open status = 0
+find_message status = 0
+found message = 1
+filename = MAIL_DIR/cur/41:2,
+== stderr ==
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
test_done
--
2.7.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH 6/6] WIP: support XDG database directory
2016-03-12 12:31 ` [PATCH 6/6] WIP: support XDG database directory David Bremner
@ 2016-03-14 18:06 ` Tomi Ollila
2016-03-19 20:19 ` David Bremner
0 siblings, 1 reply; 15+ messages in thread
From: Tomi Ollila @ 2016-03-14 18:06 UTC (permalink / raw)
To: David Bremner, notmuch
This is good opening for (eventually "fixing" e.g. library interface...);
in its current state I can come up 2 (easily solvable) problems
1) the xapian database holding email indexes can grow to be quite large;
user may have large space for emails (somewhere else than HOME) but small
HOME. This can be "fixed" by keeping the database still in the same hierarchy
as email files. Making user to point XDG_DATA_HOME elsewhere is (I) global
and (II) fragile for an user to do.
2) User may have multiple separate email configurations under one user
account; This is easy to fix with environment variable; if exists,
overrides XDG_DATA_HOME or $HOME/.local/share -- actually if this holds
"only" configurations, the XDG_CONFIG_HOME is more appropriate
(again, in simple case users should not be bothered to set any environment
variables themselves, but should be able to survive with large databases).
(standard disclaimer apply :)
Tomi
PS: I have not (yet) looked the other patches; been too
busy writing code to my personal htpc environment...
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH 6/6] WIP: support XDG database directory
2016-03-14 18:06 ` Tomi Ollila
@ 2016-03-19 20:19 ` David Bremner
0 siblings, 0 replies; 15+ messages in thread
From: David Bremner @ 2016-03-19 20:19 UTC (permalink / raw)
To: Tomi Ollila, notmuch
Tomi Ollila <tomi.ollila@iki.fi> writes:
> This is good opening for (eventually "fixing" e.g. library interface...);
> in its current state I can come up 2 (easily solvable) problems
>
> 1) the xapian database holding email indexes can grow to be quite large;
> user may have large space for emails (somewhere else than HOME) but small
> HOME. This can be "fixed" by keeping the database still in the same hierarchy
> as email files. Making user to point XDG_DATA_HOME elsewhere is (I) global
> and (II) fragile for an user to do.
Sure, once we find the database, the mail tree can be anywhere,
including the same place. The question is how do non-CLI clients find
the database. I agree that setting XDG_DATA_HOME just for notmuch sounds
ick. As you mention below we can fairly easily have our own environment
variable (NOTMUCH_DATABASE ?) that is checked in preference to XDG_DATA_HOME.
> 2) User may have multiple separate email configurations under one user
> account; This is easy to fix with environment variable; if exists,
> overrides XDG_DATA_HOME or $HOME/.local/share -- actually if this holds
> "only" configurations, the XDG_CONFIG_HOME is more appropriate
> (again, in simple case users should not be bothered to set any environment
> variables themselves, but should be able to survive with large databases).
For me, having an environment variable makes sense in terms of getting
various tools (scripts etc...) to cooperate. For multiple databases,
this seems less natural but of course the tools can still take some
parameter specifying a database. Also, I don't know how common it is for
people to have multiple databases.
d
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2016-03-19 20:19 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-01-23 14:59 library config API David Bremner
2016-01-23 14:59 ` [PATCH 1/5] lib: provide " David Bremner
2016-01-23 14:59 ` [PATCH 2/5] lib: config list iterators David Bremner
2016-01-23 14:59 ` [PATCH 3/5] CLI: add print_status_database David Bremner
2016-01-23 14:59 ` [PATCH 4/5] CLI: add optional config data to dump output David Bremner
2016-01-23 14:59 ` [PATCH 5/5] CLI: optionally restore config data David Bremner
2016-03-12 12:31 ` library config api v4 David Bremner
2016-03-12 12:31 ` [PATCH 1/6] lib: provide config API David Bremner
2016-03-12 12:31 ` [PATCH 2/6] lib: config list iterators David Bremner
2016-03-12 12:31 ` [PATCH 3/6] CLI: add print_status_database David Bremner
2016-03-12 12:31 ` [PATCH 4/6] CLI: add optional config data to dump output David Bremner
2016-03-12 12:31 ` [PATCH 5/6] CLI: optionally restore config data David Bremner
2016-03-12 12:31 ` [PATCH 6/6] WIP: support XDG database directory David Bremner
2016-03-14 18:06 ` Tomi Ollila
2016-03-19 20:19 ` David Bremner
Code repositories for project(s) associated with this public inbox
https://yhetil.org/notmuch.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).