* [RFC PATCH 00/13] Modular message store code
@ 2012-02-15 22:01 Ethan Glasser-Camp
2012-02-15 22:01 ` [RFC PATCH 01/13] Create configuration paramater database.type Ethan Glasser-Camp
` (14 more replies)
0 siblings, 15 replies; 17+ messages in thread
From: Ethan Glasser-Camp @ 2012-02-15 22:01 UTC (permalink / raw)
To: notmuch
Hi guys,
I'm submitting as RFC this patch series, which introduces the idea of a "mailstore", a "class" that defines how to access mail, instead of currently assuming it's always some Maildir-ish hierarchy that contains a bunch of mail.
This was listed as a wishlist item on http://notmuchmail.org/feature-requests/, so I went ahead and took a crack at it, learning a lot about the codebase as I did. I'm sure there are tons of stylistic concerns so I'd like to get as much feedback as possible, starting of course with "Does a feature like this have a chance of ever making it in" and followed by "Am I on the right track".
Note that this series breaks the language bundings; the Python bindings have very minimal tests so I very minimally fixed them (probably still broken in other ways), but the Ruby and Go bindings are probably super broken. Note also that the one message = one file approach is pretty thoroughly embedded into Notmuch and there are lots of places (again such as the Python bindings) where this has yet to be rooted out.
They say an interface isn't trustworthy until you've implemented it three times, so while most of the patches define the interface, the last patch adds support for an experimental CouchDB backend. It's got at least one known bug (it indexes everything, whether or not it's a mail object), sometimes it hangs when trying to access a message, and it's definitely leaking memory in notmuch-new. I haven't done strict timing comparisons but it seems like notmuch-search and notmuch-show are approximately as fast as with maildir and notmuch-new takes maybe 25% longer. Nevertheless, it is included as a demonstration that the interface is at least plausible.
The implementation of "mailstores" follows these principles:
- Messages still have "filenames", but the mailstore gets to define its own semantics for these filenames (document ids, file + byte offset..). _notmuch_message_ensure_filename_list converts all filenames coming out of the DB to be absolute paths centered at the user's database path, which is inconvenient for Couchdb, but workable.
- The user keeps all mail in one mailstore. The alternative, which is that each message can be in a different mailstore, seemed like a lot more work. "One mailstore" makes sense when it's cases like maildir vs. couchdb, but if we decide to introduce a "mbox" backend -- directory tree with mbox files -- then it might "overlap" with the maildir mailstore, and then who knows?
Patch 1 introduces the configuration parameter database.type, which will be used to select a mailstore type.
Patch 2 introduces the most important API for a mailstore, notmuch_mailstore_open, and makes it required when creating a message_file. Patch 3 fixes the Python breakage this creates.
Patch 4-6 replace the other places where files are opened directly with calls to notmuch_mailstore_open.
Patch 7-8 prepare notmuch-new to be more generic. I couldn't find an elegant way to combine add_files logic with other mailstores, so I just decided each mailstore should have its own add_files function.
Patches 9-11 add other functions to the mailstore API -- to rename files, to close files, and to "construct" a mailstore. Patch 12 uses the "close" API to close files (where necessary).
Patch 13 proposes the CouchDB mailstore as one block of code.
Points to address:
- Where to put the new notmuch_mailstore_t* parameter in all these functions? I applied a "decreasing order of importance" heuristic, but it's a little weird in places like notmuch_database_open and notmuch_database_create.
- Is there a better, more elegant way to pass around mailstore objects without adding parameters to each function? Additionally, should I be using some other class-like mechanism for mailstores instead of hacks involving structs?
- Should this be configured under [database] or perhaps under some other new heading?
- How strict is the rule that braces shouldn't be there if the body of a loop/conditional is only one line? This feels really strange to me coming from Python.
- If I'm already touching code, should I add other drive-by fixes, as in patch 05, or should I resolutely refuse to change anything, as in patch 07?
- Should something like the CouchDB backend be optional, and if so, what mechanisms do I need to use to make that happen?
Thanks so much for your time!
Ethan
^ permalink raw reply [flat|nested] 17+ messages in thread
* [RFC PATCH 01/13] Create configuration paramater database.type
2012-02-15 22:01 [RFC PATCH 00/13] Modular message store code Ethan Glasser-Camp
@ 2012-02-15 22:01 ` Ethan Glasser-Camp
2012-02-15 22:01 ` [RFC PATCH 02/13] Add the concept of a mailstore in its absolute minimal sense Ethan Glasser-Camp
` (13 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Ethan Glasser-Camp @ 2012-02-15 22:01 UTC (permalink / raw)
To: notmuch; +Cc: Ethan Glasser-Camp
From: Ethan Glasser-Camp <ethan@betacantrips.com>
This will be used to allow different backends to be developed to allow
access to mail that isn't stored in Maildirs.
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
---
notmuch-client.h | 7 ++++++
notmuch-config.c | 57 +++++++++++++++++++++++++++++++++++++++++++++++++----
2 files changed, 59 insertions(+), 5 deletions(-)
diff --git a/notmuch-client.h b/notmuch-client.h
index 60828aa..4518cb0 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -220,6 +220,13 @@ notmuch_config_set_database_path (notmuch_config_t *config,
const char *database_path);
const char *
+notmuch_config_get_database_type (notmuch_config_t *config);
+
+void
+notmuch_config_set_database_type (notmuch_config_t *config,
+ const char *database_type);
+
+const char *
notmuch_config_get_user_name (notmuch_config_t *config);
void
diff --git a/notmuch-config.c b/notmuch-config.c
index a124e34..b8bee69 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -32,11 +32,24 @@ static const char toplevel_config_comment[] =
static const char database_config_comment[] =
" Database configuration\n"
"\n"
- " The only value supported here is 'path' which should be the top-level\n"
- " directory where your mail currently exists and to where mail will be\n"
- " delivered in the future. Files should be individual email messages.\n"
- " Notmuch will store its database within a sub-directory of the path\n"
- " configured here named \".notmuch\".\n";
+ " Here is where you can tell notmuch where your mail currently exists\n"
+ " and where mail will be delivered in the future."
+ "\n"
+ " The following options are supported here:\n"
+ "\n"
+ "\ttype The type of mail backend. The only currently supported\n"
+ "\t value is \"maildir\".\n"
+ "\tpath For the maildir backend, the top-level maildir directory.\n"
+ "\t For all backends, the location where notmuch should store its\n"
+ "\t database. Notmuch will store its database within a sub-directory\n"
+ "\t of this path named \".notmuch\".\n"
+ "\n"
+ " Maildir backend\n"
+ "\n"
+ " This backend reads mail from a directory tree where files are\n"
+ " individual email messages.\n"
+ " The only configuration option is 'path' which should be the top-level\n"
+ " directory.\n";
static const char new_config_comment[] =
" Configuration for \"notmuch new\"\n"
@@ -99,6 +112,7 @@ struct _notmuch_config {
GKeyFile *key_file;
char *database_path;
+ char *database_type;
char *user_name;
char *user_primary_email;
const char **user_other_email;
@@ -258,6 +272,7 @@ notmuch_config_open (void *ctx,
config->key_file = g_key_file_new ();
config->database_path = NULL;
+ config->database_type = NULL;
config->user_name = NULL;
config->user_primary_email = NULL;
config->user_other_email = NULL;
@@ -320,6 +335,10 @@ notmuch_config_open (void *ctx,
talloc_free (path);
}
+ if (notmuch_config_get_database_type (config) == NULL) {
+ notmuch_config_set_database_type (config, "maildir");
+ }
+
if (notmuch_config_get_user_name (config) == NULL) {
char *name = get_name_from_passwd_file (config);
notmuch_config_set_user_name (config, name);
@@ -538,6 +557,34 @@ notmuch_config_set_database_path (notmuch_config_t *config,
}
const char *
+notmuch_config_get_database_type (notmuch_config_t *config)
+{
+ char *type;
+
+ if (config->database_type == NULL) {
+ type = g_key_file_get_string (config->key_file,
+ "database", "type", NULL);
+ if (type) {
+ config->database_type = talloc_strdup (config, type);
+ free (type);
+ }
+ }
+
+ return config->database_type;
+}
+
+void
+notmuch_config_set_database_type (notmuch_config_t *config,
+ const char *database_type)
+{
+ g_key_file_set_string (config->key_file,
+ "database", "type", database_type);
+
+ talloc_free (config->database_type);
+ config->database_type = NULL;
+}
+
+const char *
notmuch_config_get_user_name (notmuch_config_t *config)
{
char *name;
--
1.7.5.4
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [RFC PATCH 02/13] Add the concept of a mailstore in its absolute minimal sense
2012-02-15 22:01 [RFC PATCH 00/13] Modular message store code Ethan Glasser-Camp
2012-02-15 22:01 ` [RFC PATCH 01/13] Create configuration paramater database.type Ethan Glasser-Camp
@ 2012-02-15 22:01 ` Ethan Glasser-Camp
2012-02-15 22:01 ` [RFC PATCH 03/13] Introduce mailstore in the python bindings Ethan Glasser-Camp
` (12 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Ethan Glasser-Camp @ 2012-02-15 22:01 UTC (permalink / raw)
To: notmuch; +Cc: Ethan Glasser-Camp
From: Ethan Glasser-Camp <ethan@betacantrips.com>
This introduces (and uses) the mailstore parameter to the
notmuch_message_file_open API, and passes this through wherever it
will be needed. This requires touching a lot of places just to change
one API. We end up adding it to the notmuch_database_t struct because
it is needed for notmuch_database_upgrade.
This doesn't touch the Python bindings, which require a certain amount
of effort. (Therefore, the Python tests will be broken until the next commit.)
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
---
lib/Makefile.local | 1 +
lib/database-private.h | 1 +
lib/database.cc | 10 +++++---
lib/mailstore.c | 59 ++++++++++++++++++++++++++++++++++++++++++++++++
lib/message-file.c | 10 +++++---
lib/message.cc | 11 +++++---
lib/notmuch-private.h | 7 ++++-
lib/notmuch.h | 21 ++++++++++++++--
lib/thread.cc | 24 +++++++++++--------
notmuch-client.h | 6 +++++
notmuch-config.c | 12 +++++++++
notmuch-count.c | 3 +-
notmuch-dump.c | 3 +-
notmuch-new.c | 8 +++++-
notmuch-reply.c | 45 ++++++++++++++++++++++++------------
notmuch-restore.c | 3 +-
notmuch-search.c | 3 +-
notmuch-show.c | 56 ++++++++++++++++++++++++++++++---------------
notmuch-tag.c | 3 +-
test/symbol-test.cc | 3 +-
20 files changed, 220 insertions(+), 69 deletions(-)
create mode 100644 lib/mailstore.c
diff --git a/lib/Makefile.local b/lib/Makefile.local
index 54c4dea..461e5cd 100644
--- a/lib/Makefile.local
+++ b/lib/Makefile.local
@@ -51,6 +51,7 @@ libnotmuch_c_srcs = \
$(dir)/filenames.c \
$(dir)/string-list.c \
$(dir)/libsha1.c \
+ $(dir)/mailstore.c \
$(dir)/message-file.c \
$(dir)/messages.c \
$(dir)/sha1.c \
diff --git a/lib/database-private.h b/lib/database-private.h
index 88532d5..1cb8c43 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -39,6 +39,7 @@
struct _notmuch_database {
notmuch_bool_t exception_reported;
+ notmuch_mailstore_t *mailstore;
char *path;
notmuch_bool_t needs_upgrade;
diff --git a/lib/database.cc b/lib/database.cc
index c928d02..e3c8095 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -521,7 +521,7 @@ parse_references (void *ctx,
}
notmuch_database_t *
-notmuch_database_create (const char *path)
+notmuch_database_create (notmuch_mailstore_t *mailstore, const char *path)
{
notmuch_database_t *notmuch = NULL;
char *notmuch_path = NULL;
@@ -556,7 +556,7 @@ notmuch_database_create (const char *path)
goto DONE;
}
- notmuch = notmuch_database_open (path,
+ notmuch = notmuch_database_open (mailstore, path,
NOTMUCH_DATABASE_MODE_READ_WRITE);
notmuch_database_upgrade (notmuch, NULL, NULL);
@@ -579,7 +579,8 @@ _notmuch_database_ensure_writable (notmuch_database_t *notmuch)
}
notmuch_database_t *
-notmuch_database_open (const char *path,
+notmuch_database_open (notmuch_mailstore_t *mailstore,
+ const char *path,
notmuch_database_mode_t mode)
{
void *local = talloc_new (NULL);
@@ -619,6 +620,7 @@ notmuch_database_open (const char *path,
notmuch = talloc_zero (NULL, notmuch_database_t);
notmuch->exception_reported = FALSE;
notmuch->path = talloc_strdup (notmuch, path);
+ notmuch->mailstore = mailstore;
if (notmuch->path[strlen (notmuch->path) - 1] == '/')
notmuch->path[strlen (notmuch->path) - 1] = '\0';
@@ -1636,7 +1638,7 @@ notmuch_database_add_message (notmuch_database_t *notmuch,
if (ret)
return ret;
- message_file = notmuch_message_file_open (filename);
+ message_file = notmuch_message_file_open (notmuch->mailstore, filename);
if (message_file == NULL)
return NOTMUCH_STATUS_FILE_ERROR;
diff --git a/lib/mailstore.c b/lib/mailstore.c
new file mode 100644
index 0000000..290da70
--- /dev/null
+++ b/lib/mailstore.c
@@ -0,0 +1,59 @@
+/* mailstore.c - mail storage backends
+ *
+ * Copyright © 2009 Carl Worth
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see http://www.gnu.org/licenses/ .
+ */
+
+#include <stdio.h>
+
+#include "notmuch-private.h"
+
+typedef struct _notmuch_mailstore {
+ FILE *(*open) (struct _notmuch_mailstore *mailstore, const char *filename);
+} _notmuch_mailstore;
+
+static FILE *
+_maildir_open_function (unused (notmuch_mailstore_t *mailstore),
+ const char *filename)
+{
+ return fopen (filename, "r");
+}
+
+/* A mailstore is defined as:
+ *
+ * - A function used to "open" a mail message. This takes the
+ * "filename" for the file and should return a FILE *.
+ *
+ * - TODO: A way to scan for new messages?
+ *
+ * - TODO: A "constructor"?
+ */
+_notmuch_mailstore
+notmuch_mailstore_maildir = { _maildir_open_function };
+
+_notmuch_mailstore *
+notmuch_mailstore_get_by_name (const char *name)
+{
+ if (strcmp (name, "maildir") == 0)
+ return ¬much_mailstore_maildir;
+
+ return NULL;
+}
+
+FILE *
+notmuch_mailstore_open (notmuch_mailstore_t *mailstore, const char *filename)
+{
+ return mailstore->open (mailstore, filename);
+}
diff --git a/lib/message-file.c b/lib/message-file.c
index 915aba8..61f4d04 100644
--- a/lib/message-file.c
+++ b/lib/message-file.c
@@ -94,7 +94,8 @@ _notmuch_message_file_destructor (notmuch_message_file_t *message)
/* Create a new notmuch_message_file_t for 'filename' with 'ctx' as
* the talloc owner. */
notmuch_message_file_t *
-_notmuch_message_file_open_ctx (void *ctx, const char *filename)
+_notmuch_message_file_open_ctx (void *ctx, notmuch_mailstore_t *mailstore,
+ const char *filename)
{
notmuch_message_file_t *message;
@@ -104,7 +105,7 @@ _notmuch_message_file_open_ctx (void *ctx, const char *filename)
talloc_set_destructor (message, _notmuch_message_file_destructor);
- message->file = fopen (filename, "r");
+ message->file = notmuch_mailstore_open (mailstore, filename);
if (message->file == NULL)
goto FAIL;
@@ -126,9 +127,10 @@ _notmuch_message_file_open_ctx (void *ctx, const char *filename)
}
notmuch_message_file_t *
-notmuch_message_file_open (const char *filename)
+notmuch_message_file_open (notmuch_mailstore_t *mailstore,
+ const char *filename)
{
- return _notmuch_message_file_open_ctx (NULL, filename);
+ return _notmuch_message_file_open_ctx (NULL, mailstore, filename);
}
void
diff --git a/lib/message.cc b/lib/message.cc
index 0075425..762a18f 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -395,7 +395,8 @@ notmuch_message_get_message_id (notmuch_message_t *message)
}
static void
-_notmuch_message_ensure_message_file (notmuch_message_t *message)
+_notmuch_message_ensure_message_file (notmuch_mailstore_t *mailstore,
+ notmuch_message_t *message)
{
const char *filename;
@@ -406,11 +407,13 @@ _notmuch_message_ensure_message_file (notmuch_message_t *message)
if (unlikely (filename == NULL))
return;
- message->message_file = _notmuch_message_file_open_ctx (message, filename);
+ message->message_file = _notmuch_message_file_open_ctx (message, mailstore,
+ filename);
}
const char *
-notmuch_message_get_header (notmuch_message_t *message, const char *header)
+notmuch_message_get_header (notmuch_mailstore_t *mailstore,
+ notmuch_message_t *message, const char *header)
{
std::string value;
@@ -427,7 +430,7 @@ notmuch_message_get_header (notmuch_message_t *message, const char *header)
return talloc_strdup (message, value.c_str ());
/* Otherwise fall back to parsing the file */
- _notmuch_message_ensure_message_file (message);
+ _notmuch_message_ensure_message_file (mailstore, message);
if (message->message_file == NULL)
return NULL;
diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index 7bf153e..0f01437 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -331,11 +331,14 @@ typedef struct _notmuch_message_file notmuch_message_file_t;
* Returns NULL if any error occurs.
*/
notmuch_message_file_t *
-notmuch_message_file_open (const char *filename);
+notmuch_message_file_open (notmuch_mailstore_t *mailstore,
+ const char *filename);
/* Like notmuch_message_file_open but with 'ctx' as the talloc owner. */
notmuch_message_file_t *
-_notmuch_message_file_open_ctx (void *ctx, const char *filename);
+_notmuch_message_file_open_ctx (void *ctx,
+ notmuch_mailstore_t *mailstore,
+ const char *filename);
/* Close a notmuch message previously opened with notmuch_message_open. */
void
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 7929fe7..7ebe034 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -123,6 +123,7 @@ typedef struct _notmuch_message notmuch_message_t;
typedef struct _notmuch_tags notmuch_tags_t;
typedef struct _notmuch_directory notmuch_directory_t;
typedef struct _notmuch_filenames notmuch_filenames_t;
+typedef struct _notmuch_mailstore notmuch_mailstore_t;
/* Create a new, empty notmuch database located at 'path'.
*
@@ -144,7 +145,7 @@ typedef struct _notmuch_filenames notmuch_filenames_t;
* an error message on stderr).
*/
notmuch_database_t *
-notmuch_database_create (const char *path);
+notmuch_database_create (notmuch_mailstore_t *mailstore, const char *path);
typedef enum {
NOTMUCH_DATABASE_MODE_READ_ONLY = 0,
@@ -172,7 +173,8 @@ typedef enum {
* an error message on stderr).
*/
notmuch_database_t *
-notmuch_database_open (const char *path,
+notmuch_database_open (notmuch_mailstore_t *mailstore,
+ const char *path,
notmuch_database_mode_t mode);
/* Close the given notmuch database, freeing all associated
@@ -409,6 +411,18 @@ notmuch_database_find_message_by_filename (notmuch_database_t *notmuch,
notmuch_tags_t *
notmuch_database_get_all_tags (notmuch_database_t *db);
+/* Return a mail storage backend based on the given name.
+ *
+ * Storage backends are required in order to manipulate message files.
+ */
+notmuch_mailstore_t *
+notmuch_mailstore_get_by_name (const char *name);
+
+/* Get an input stream for a filename.
+ */
+FILE *
+notmuch_mailstore_open (notmuch_mailstore_t *mailstore, const char *filename);
+
/* Create a new query for 'database'.
*
* Here, 'database' should be an open database, (see
@@ -929,7 +943,8 @@ notmuch_message_get_date (notmuch_message_t *message);
* header line matching 'header'. Returns NULL if any error occurs.
*/
const char *
-notmuch_message_get_header (notmuch_message_t *message, const char *header);
+notmuch_message_get_header (notmuch_mailstore_t *mailstore,
+ notmuch_message_t *message, const char *header);
/* Get the tags for 'message', returning a notmuch_tags_t object which
* can be used to iterate over all tags.
diff --git a/lib/thread.cc b/lib/thread.cc
index 0435ee6..73edf83 100644
--- a/lib/thread.cc
+++ b/lib/thread.cc
@@ -213,7 +213,8 @@ _thread_cleanup_author (notmuch_thread_t *thread,
* reference to it.
*/
static void
-_thread_add_message (notmuch_thread_t *thread,
+_thread_add_message (notmuch_mailstore_t *mailstore,
+ notmuch_thread_t *thread,
notmuch_message_t *message)
{
notmuch_tags_t *tags;
@@ -231,7 +232,7 @@ _thread_add_message (notmuch_thread_t *thread,
xstrdup (notmuch_message_get_message_id (message)),
message);
- from = notmuch_message_get_header (message, "from");
+ from = notmuch_message_get_header (mailstore, message, "from");
if (from)
list = internet_address_list_parse_string (from);
@@ -253,7 +254,7 @@ _thread_add_message (notmuch_thread_t *thread,
if (! thread->subject) {
const char *subject;
- subject = notmuch_message_get_header (message, "subject");
+ subject = notmuch_message_get_header (mailstore, message, "subject");
thread->subject = talloc_strdup (thread, subject ? subject : "");
}
@@ -267,13 +268,14 @@ _thread_add_message (notmuch_thread_t *thread,
}
static void
-_thread_set_subject_from_message (notmuch_thread_t *thread,
+_thread_set_subject_from_message (notmuch_mailstore_t *mailstore,
+ notmuch_thread_t *thread,
notmuch_message_t *message)
{
const char *subject;
const char *cleaned_subject;
- subject = notmuch_message_get_header (message, "subject");
+ subject = notmuch_message_get_header (mailstore, message, "subject");
if (! subject)
return;
@@ -300,7 +302,8 @@ _thread_set_subject_from_message (notmuch_thread_t *thread,
* oldest or newest matching subject is applied to the thread as a
* whole. */
static void
-_thread_add_matched_message (notmuch_thread_t *thread,
+_thread_add_matched_message (notmuch_mailstore_t *mailstore,
+ notmuch_thread_t *thread,
notmuch_message_t *message,
notmuch_sort_t sort)
{
@@ -312,13 +315,13 @@ _thread_add_matched_message (notmuch_thread_t *thread,
if (date < thread->oldest || ! thread->matched_messages) {
thread->oldest = date;
if (sort == NOTMUCH_SORT_OLDEST_FIRST)
- _thread_set_subject_from_message (thread, message);
+ _thread_set_subject_from_message (mailstore, thread, message);
}
if (date > thread->newest || ! thread->matched_messages) {
thread->newest = date;
if (sort != NOTMUCH_SORT_OLDEST_FIRST)
- _thread_set_subject_from_message (thread, message);
+ _thread_set_subject_from_message (mailstore, thread, message);
}
thread->matched_messages++;
@@ -467,11 +470,12 @@ _notmuch_thread_create (void *ctx,
if (doc_id == seed_doc_id)
message = seed_message;
- _thread_add_message (thread, message);
+ _thread_add_message (notmuch->mailstore, thread, message);
if ( _notmuch_doc_id_set_contains (match_set, doc_id)) {
_notmuch_doc_id_set_remove (match_set, doc_id);
- _thread_add_matched_message (thread, message, sort);
+ _thread_add_matched_message (notmuch->mailstore, thread,
+ message, sort);
}
_notmuch_message_close (message);
diff --git a/notmuch-client.h b/notmuch-client.h
index 4518cb0..c1c30a2 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -68,14 +68,17 @@ struct notmuch_show_params;
typedef struct notmuch_show_format {
const char *message_set_start;
void (*part) (const void *ctx,
+ notmuch_mailstore_t *mailstore,
struct mime_node *node, int indent,
const struct notmuch_show_params *params);
const char *message_start;
void (*message) (const void *ctx,
+ notmuch_mailstore_t *mailstore,
notmuch_message_t *message,
int indent);
const char *header_start;
void (*header) (const void *ctx,
+ notmuch_mailstore_t *mailstore,
notmuch_message_t *message);
void (*header_message_part) (GMimeMessage *message);
const char *header_end;
@@ -226,6 +229,9 @@ void
notmuch_config_set_database_type (notmuch_config_t *config,
const char *database_type);
+notmuch_mailstore_t *
+notmuch_config_get_mailstore (notmuch_config_t *config);
+
const char *
notmuch_config_get_user_name (notmuch_config_t *config);
diff --git a/notmuch-config.c b/notmuch-config.c
index b8bee69..f611b26 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -584,6 +584,18 @@ notmuch_config_set_database_type (notmuch_config_t *config,
config->database_type = NULL;
}
+notmuch_mailstore_t *
+notmuch_config_get_mailstore (notmuch_config_t *config)
+{
+ /* This is just a stub since there's only one mailstore.
+ *
+ * When there are multiple mailstore types and "constructors" for
+ * them, this may have to be much more complicated.
+ */
+ const char *type = notmuch_config_get_database_type (config);
+ return notmuch_mailstore_get_by_name (type);
+}
+
const char *
notmuch_config_get_user_name (notmuch_config_t *config)
{
diff --git a/notmuch-count.c b/notmuch-count.c
index 63459fb..0c7beef 100644
--- a/notmuch-count.c
+++ b/notmuch-count.c
@@ -57,7 +57,8 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
if (config == NULL)
return 1;
- notmuch = notmuch_database_open (notmuch_config_get_database_path (config),
+ notmuch = notmuch_database_open (notmuch_config_get_mailstore (config),
+ notmuch_config_get_database_path (config),
NOTMUCH_DATABASE_MODE_READ_ONLY);
if (notmuch == NULL)
return 1;
diff --git a/notmuch-dump.c b/notmuch-dump.c
index a735875..f7729dd 100644
--- a/notmuch-dump.c
+++ b/notmuch-dump.c
@@ -36,7 +36,8 @@ notmuch_dump_command (unused (void *ctx), int argc, char *argv[])
if (config == NULL)
return 1;
- notmuch = notmuch_database_open (notmuch_config_get_database_path (config),
+ notmuch = notmuch_database_open (notmuch_config_get_mailstore (config),
+ notmuch_config_get_database_path (config),
NOTMUCH_DATABASE_MODE_READ_ONLY);
if (notmuch == NULL)
return 1;
diff --git a/notmuch-new.c b/notmuch-new.c
index 8dbebb3..355dde4 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -808,6 +808,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
{
notmuch_config_t *config;
notmuch_database_t *notmuch;
+ notmuch_mailstore_t *mailstore;
add_files_state_t add_files_state;
double elapsed;
struct timeval tv_now, tv_start;
@@ -843,6 +844,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
add_files_state.new_tags = notmuch_config_get_new_tags (config, &add_files_state.new_tags_length);
add_files_state.synchronize_flags = notmuch_config_get_maildir_synchronize_flags (config);
db_path = notmuch_config_get_database_path (config);
+ mailstore = notmuch_config_get_mailstore (config);
if (run_hooks) {
ret = notmuch_run_hook (db_path, "pre-new");
@@ -861,10 +863,12 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
return 1;
printf ("Found %d total files (that's not much mail).\n", count);
- notmuch = notmuch_database_create (db_path);
+ notmuch = notmuch_database_create (mailstore,
+ db_path);
add_files_state.total_files = count;
} else {
- notmuch = notmuch_database_open (db_path,
+ notmuch = notmuch_database_open (mailstore,
+ db_path,
NOTMUCH_DATABASE_MODE_READ_WRITE);
if (notmuch == NULL)
return 1;
diff --git a/notmuch-reply.c b/notmuch-reply.c
index 6b244e6..cb1dd6e 100644
--- a/notmuch-reply.c
+++ b/notmuch-reply.c
@@ -256,14 +256,15 @@ scan_address_string (const char *recipients,
* in either the 'To' or 'Cc' header of the message?
*/
static int
-reply_to_header_is_redundant (notmuch_message_t *message)
+reply_to_header_is_redundant (notmuch_mailstore_t *mailstore,
+ notmuch_message_t *message)
{
const char *reply_to, *to, *cc, *addr;
InternetAddressList *list;
InternetAddress *address;
InternetAddressMailbox *mailbox;
- reply_to = notmuch_message_get_header (message, "reply-to");
+ reply_to = notmuch_message_get_header (mailstore, message, "reply-to");
if (reply_to == NULL || *reply_to == '\0')
return 0;
@@ -279,8 +280,8 @@ reply_to_header_is_redundant (notmuch_message_t *message)
mailbox = INTERNET_ADDRESS_MAILBOX (address);
addr = internet_address_mailbox_get_addr (mailbox);
- to = notmuch_message_get_header (message, "to");
- cc = notmuch_message_get_header (message, "cc");
+ to = notmuch_message_get_header (mailstore, message, "to");
+ cc = notmuch_message_get_header (mailstore, message, "cc");
if ((to && strstr (to, addr) != 0) ||
(cc && strstr (cc, addr) != 0))
@@ -319,10 +320,13 @@ add_recipients_from_message (GMimeMessage *reply,
{ "cc", NULL, GMIME_RECIPIENT_TYPE_CC },
{ "bcc", NULL, GMIME_RECIPIENT_TYPE_BCC }
};
+ notmuch_mailstore_t *mailstore;
const char *from_addr = NULL;
unsigned int i;
unsigned int n = 0;
+ mailstore = notmuch_config_get_mailstore (config);
+
/* Some mailing lists munge the Reply-To header despite it being A Bad
* Thing, see http://www.unicom.com/pw/reply-to-harmful.html
*
@@ -334,7 +338,7 @@ add_recipients_from_message (GMimeMessage *reply,
* that the address in the Reply-To header will always appear in
* the reply.
*/
- if (reply_to_header_is_redundant (message)) {
+ if (reply_to_header_is_redundant (mailstore, message)) {
reply_to_map[0].header = "from";
reply_to_map[0].fallback = NULL;
}
@@ -342,10 +346,10 @@ add_recipients_from_message (GMimeMessage *reply,
for (i = 0; i < ARRAY_SIZE (reply_to_map); i++) {
const char *recipients;
- recipients = notmuch_message_get_header (message,
+ recipients = notmuch_message_get_header (mailstore, message,
reply_to_map[i].header);
if ((recipients == NULL || recipients[0] == '\0') && reply_to_map[i].fallback)
- recipients = notmuch_message_get_header (message,
+ recipients = notmuch_message_get_header (mailstore, message,
reply_to_map[i].fallback);
n += scan_address_string (recipients, config, reply,
@@ -374,6 +378,7 @@ add_recipients_from_message (GMimeMessage *reply,
static const char *
guess_from_received_header (notmuch_config_t *config, notmuch_message_t *message)
{
+ notmuch_mailstore_t *mailstore;
const char *received,*primary,*by;
const char **other;
char *tohdr;
@@ -387,6 +392,7 @@ guess_from_received_header (notmuch_config_t *config, notmuch_message_t *message
primary = notmuch_config_get_user_primary_email (config);
other = notmuch_config_get_user_other_email (config, &other_len);
+ mailstore = notmuch_config_get_mailstore (config);
/* sadly, there is no standard way to find out to which email
* address a mail was delivered - what is in the headers depends
@@ -403,7 +409,8 @@ guess_from_received_header (notmuch_config_t *config, notmuch_message_t *message
* If none of these work, we give up and return NULL
*/
for (i = 0; i < sizeof(to_headers)/sizeof(*to_headers); i++) {
- tohdr = xstrdup(notmuch_message_get_header (message, to_headers[i]));
+ tohdr = xstrdup(notmuch_message_get_header (mailstore,
+ message, to_headers[i]));
if (tohdr && *tohdr) {
/* tohdr is potentialy a list of email addresses, so here we
* check if one of the email addresses is a substring of tohdr
@@ -428,7 +435,7 @@ guess_from_received_header (notmuch_config_t *config, notmuch_message_t *message
* The Received: header is special in our get_header function
* and is always concatenated.
*/
- received = notmuch_message_get_header (message, "received");
+ received = notmuch_message_get_header (mailstore, message, "received");
if (received == NULL)
return NULL;
@@ -515,10 +522,12 @@ notmuch_reply_format_default(void *ctx,
GMimeMessage *reply;
notmuch_messages_t *messages;
notmuch_message_t *message;
+ notmuch_mailstore_t *mailstore;
const char *subject, *from_addr = NULL;
const char *in_reply_to, *orig_references, *references;
const notmuch_show_format_t *format = &format_reply;
+ mailstore = notmuch_config_get_mailstore (config);
for (messages = notmuch_query_search_messages (query);
notmuch_messages_valid (messages);
notmuch_messages_move_to_next (messages))
@@ -532,7 +541,7 @@ notmuch_reply_format_default(void *ctx,
return 1;
}
- subject = notmuch_message_get_header (message, "subject");
+ subject = notmuch_message_get_header (mailstore, message, "subject");
if (subject) {
if (strncasecmp (subject, "Re:", 3))
subject = talloc_asprintf (ctx, "Re: %s", subject);
@@ -560,7 +569,8 @@ notmuch_reply_format_default(void *ctx,
g_mime_object_set_header (GMIME_OBJECT (reply),
"In-Reply-To", in_reply_to);
- orig_references = notmuch_message_get_header (message, "references");
+ orig_references = notmuch_message_get_header (mailstore,
+ message, "references");
references = talloc_asprintf (ctx, "%s%s%s",
orig_references ? orig_references : "",
orig_references ? " " : "",
@@ -574,8 +584,8 @@ notmuch_reply_format_default(void *ctx,
reply = NULL;
printf ("On %s, %s wrote:\n",
- notmuch_message_get_header (message, "date"),
- notmuch_message_get_header (message, "from"));
+ notmuch_message_get_header (mailstore, message, "date"),
+ notmuch_message_get_header (mailstore, message, "from"));
show_message_body (message, format, params);
@@ -595,9 +605,12 @@ notmuch_reply_format_headers_only(void *ctx,
GMimeMessage *reply;
notmuch_messages_t *messages;
notmuch_message_t *message;
+ notmuch_mailstore_t *mailstore;
const char *in_reply_to, *orig_references, *references;
char *reply_headers;
+ mailstore = notmuch_config_get_mailstore (config);
+
for (messages = notmuch_query_search_messages (query);
notmuch_messages_valid (messages);
notmuch_messages_move_to_next (messages))
@@ -618,7 +631,8 @@ notmuch_reply_format_headers_only(void *ctx,
"In-Reply-To", in_reply_to);
- orig_references = notmuch_message_get_header (message, "references");
+ orig_references = notmuch_message_get_header (mailstore, message,
+ "references");
/* We print In-Reply-To followed by References because git format-patch treats them
* specially. Git does not interpret the other headers specially
@@ -720,7 +734,8 @@ notmuch_reply_command (void *ctx, int argc, char *argv[])
return 1;
}
- notmuch = notmuch_database_open (notmuch_config_get_database_path (config),
+ notmuch = notmuch_database_open (notmuch_config_get_mailstore (config),
+ notmuch_config_get_database_path (config),
NOTMUCH_DATABASE_MODE_READ_ONLY);
if (notmuch == NULL)
return 1;
diff --git a/notmuch-restore.c b/notmuch-restore.c
index 87d9772..b382b7b 100644
--- a/notmuch-restore.c
+++ b/notmuch-restore.c
@@ -40,7 +40,8 @@ notmuch_restore_command (unused (void *ctx), int argc, char *argv[])
if (config == NULL)
return 1;
- notmuch = notmuch_database_open (notmuch_config_get_database_path (config),
+ notmuch = notmuch_database_open (notmuch_config_get_mailstore (config),
+ notmuch_config_get_database_path (config),
NOTMUCH_DATABASE_MODE_READ_WRITE);
if (notmuch == NULL)
return 1;
diff --git a/notmuch-search.c b/notmuch-search.c
index d504051..8ba3c48 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -470,7 +470,8 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
if (config == NULL)
return 1;
- notmuch = notmuch_database_open (notmuch_config_get_database_path (config),
+ notmuch = notmuch_database_open (notmuch_config_get_mailstore (config),
+ notmuch_config_get_database_path (config),
NOTMUCH_DATABASE_MODE_READ_ONLY);
if (notmuch == NULL)
return 1;
diff --git a/notmuch-show.c b/notmuch-show.c
index d930f94..81d4cf0 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -24,7 +24,8 @@ static void
format_headers_message_part_text (GMimeMessage *message);
static void
-format_part_text (const void *ctx, mime_node_t *node,
+format_part_text (const void *ctx, notmuch_mailstore_t *mailstore,
+ mime_node_t *node,
int indent, const notmuch_show_params_t *params);
static const notmuch_show_format_t format_text = {
@@ -36,10 +37,12 @@ static const notmuch_show_format_t format_text = {
static void
format_message_json (const void *ctx,
+ notmuch_mailstore_t *mailstore,
notmuch_message_t *message,
unused (int indent));
static void
format_headers_json (const void *ctx,
+ notmuch_mailstore_t *mailstore,
notmuch_message_t *message);
static void
@@ -83,6 +86,7 @@ static const notmuch_show_format_t format_json = {
static void
format_message_mbox (const void *ctx,
+ notmuch_mailstore_t *mailstore,
notmuch_message_t *message,
unused (int indent));
@@ -149,14 +153,15 @@ _get_tags_as_string (const void *ctx, notmuch_message_t *message)
/* Get a nice, single-line summary of message. */
static const char *
-_get_one_line_summary (const void *ctx, notmuch_message_t *message)
+_get_one_line_summary (const void *ctx, notmuch_mailstore_t *mailstore,
+ notmuch_message_t *message)
{
const char *from;
time_t date;
const char *relative_date;
const char *tags;
- from = notmuch_message_get_header (message, "from");
+ from = notmuch_message_get_header (mailstore, message, "from");
date = notmuch_message_get_date (message);
relative_date = notmuch_time_relative_date (ctx, date);
@@ -168,7 +173,8 @@ _get_one_line_summary (const void *ctx, notmuch_message_t *message)
}
static void
-format_message_json (const void *ctx, notmuch_message_t *message, unused (int indent))
+format_message_json (const void *ctx, unused (notmuch_mailstore_t *mailstore),
+ notmuch_message_t *message, unused (int indent))
{
notmuch_tags_t *tags;
int first = 1;
@@ -262,6 +268,7 @@ _is_from_line (const char *line)
*/
static void
format_message_mbox (const void *ctx,
+ notmuch_mailstore_t *mailstore,
notmuch_message_t *message,
unused (int indent))
{
@@ -285,7 +292,7 @@ format_message_mbox (const void *ctx,
return;
}
- from = notmuch_message_get_header (message, "from");
+ from = notmuch_message_get_header (mailstore, message, "from");
from = _extract_email_address (ctx, from);
date = notmuch_message_get_date (message);
@@ -327,7 +334,7 @@ format_headers_message_part_text (GMimeMessage *message)
}
static void
-format_headers_json (const void *ctx, notmuch_message_t *message)
+format_headers_json (const void *ctx, notmuch_mailstore_t *mailstore, notmuch_message_t *message)
{
const char *headers[] = {
"Subject", "From", "To", "Cc", "Bcc", "Date"
@@ -339,7 +346,7 @@ format_headers_json (const void *ctx, notmuch_message_t *message)
for (i = 0; i < ARRAY_SIZE (headers); i++) {
name = headers[i];
- value = notmuch_message_get_header (message, name);
+ value = notmuch_message_get_header (mailstore, message, name);
if (value)
{
if (!first_header)
@@ -719,7 +726,8 @@ format_part_content_raw (GMimeObject *part)
}
static void
-format_part_text (const void *ctx, mime_node_t *node,
+format_part_text (const void *ctx, notmuch_mailstore_t *mailstore,
+ mime_node_t *node,
int indent, const notmuch_show_params_t *params)
{
/* The disposition and content-type metadata are associated with
@@ -768,7 +776,8 @@ format_part_text (const void *ctx, mime_node_t *node,
printf ("\fheader{\n");
if (node->envelope_file)
- printf ("%s\n", _get_one_line_summary (ctx, node->envelope_file));
+ printf ("%s\n", _get_one_line_summary (ctx, mailstore,
+ node->envelope_file));
printf ("Subject: %s\n", g_mime_message_get_subject (message));
printf ("From: %s\n", g_mime_message_get_sender (message));
recipients = g_mime_message_get_recipients (message, GMIME_RECIPIENT_TYPE_TO);
@@ -800,7 +809,8 @@ format_part_text (const void *ctx, mime_node_t *node,
}
for (i = 0; i < node->nchildren; i++)
- format_part_text (ctx, mime_node_child (node, i), indent, params);
+ format_part_text (ctx, mailstore, mime_node_child (node, i),
+ indent, params);
if (GMIME_IS_MESSAGE (node->part))
printf ("\fbody}\n");
@@ -811,6 +821,7 @@ format_part_text (const void *ctx, mime_node_t *node,
static void
show_message (void *ctx,
const notmuch_show_format_t *format,
+ notmuch_mailstore_t *mailstore,
notmuch_message_t *message,
int indent,
notmuch_show_params_t *params)
@@ -823,7 +834,7 @@ show_message (void *ctx,
&root) == NOTMUCH_STATUS_SUCCESS &&
(part = mime_node_seek_dfs (root, (params->part < 0 ?
0 : params->part))))
- format->part (local, part, indent, params);
+ format->part (local, mailstore, part, indent, params);
talloc_free (local);
return;
}
@@ -831,11 +842,11 @@ show_message (void *ctx,
if (params->part <= 0) {
fputs (format->message_start, stdout);
if (format->message)
- format->message(ctx, message, indent);
+ format->message(ctx, mailstore, message, indent);
fputs (format->header_start, stdout);
if (format->header)
- format->header(ctx, message);
+ format->header(ctx, mailstore, message);
fputs (format->header_end, stdout);
fputs (format->body_start, stdout);
@@ -854,6 +865,7 @@ show_message (void *ctx,
static void
show_messages (void *ctx,
const notmuch_show_format_t *format,
+ notmuch_mailstore_t *mailstore,
notmuch_messages_t *messages,
int indent,
notmuch_show_params_t *params)
@@ -882,7 +894,7 @@ show_messages (void *ctx,
next_indent = indent;
if (match || params->entire_thread) {
- show_message (ctx, format, message, indent, params);
+ show_message (ctx, format, mailstore, message, indent, params);
next_indent = indent + 1;
fputs (format->message_set_sep, stdout);
@@ -890,6 +902,7 @@ show_messages (void *ctx,
show_messages (ctx,
format,
+ mailstore,
notmuch_message_get_replies (message),
next_indent,
params);
@@ -905,6 +918,7 @@ show_messages (void *ctx,
/* Formatted output of single message */
static int
do_show_single (void *ctx,
+ notmuch_mailstore_t *mailstore,
notmuch_query_t *query,
const notmuch_show_format_t *format,
notmuch_show_params_t *params)
@@ -966,7 +980,7 @@ do_show_single (void *ctx,
} else {
- show_message (ctx, format, message, 0, params);
+ show_message (ctx, format, mailstore, message, 0, params);
}
@@ -976,6 +990,7 @@ do_show_single (void *ctx,
/* Formatted output of threads */
static int
do_show (void *ctx,
+ notmuch_mailstore_t *mailstore,
notmuch_query_t *query,
const notmuch_show_format_t *format,
notmuch_show_params_t *params)
@@ -1003,7 +1018,7 @@ do_show (void *ctx,
fputs (format->message_set_sep, stdout);
first_toplevel = 0;
- show_messages (ctx, format, messages, 0, params);
+ show_messages (ctx, format, mailstore, messages, 0, params);
notmuch_thread_destroy (thread);
@@ -1027,6 +1042,7 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
{
notmuch_config_t *config;
notmuch_database_t *notmuch;
+ notmuch_mailstore_t *mailstore;
notmuch_query_t *query;
char *query_string;
int opt_index, ret;
@@ -1122,7 +1138,9 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
return 1;
}
- notmuch = notmuch_database_open (notmuch_config_get_database_path (config),
+ mailstore = notmuch_config_get_mailstore (config);
+ notmuch = notmuch_database_open (mailstore,
+ notmuch_config_get_database_path (config),
NOTMUCH_DATABASE_MODE_READ_ONLY);
if (notmuch == NULL)
return 1;
@@ -1134,9 +1152,9 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
}
if (params.part >= 0)
- ret = do_show_single (ctx, query, format, ¶ms);
+ ret = do_show_single (ctx, mailstore, query, format, ¶ms);
else
- ret = do_show (ctx, query, format, ¶ms);
+ ret = do_show (ctx, mailstore, query, format, ¶ms);
notmuch_query_destroy (query);
notmuch_database_close (notmuch);
diff --git a/notmuch-tag.c b/notmuch-tag.c
index 36b9b09..5e8d74a 100644
--- a/notmuch-tag.c
+++ b/notmuch-tag.c
@@ -187,7 +187,8 @@ notmuch_tag_command (void *ctx, int argc, char *argv[])
if (config == NULL)
return 1;
- notmuch = notmuch_database_open (notmuch_config_get_database_path (config),
+ notmuch = notmuch_database_open (notmuch_config_get_mailstore (config),
+ notmuch_config_get_database_path (config),
NOTMUCH_DATABASE_MODE_READ_WRITE);
if (notmuch == NULL)
return 1;
diff --git a/test/symbol-test.cc b/test/symbol-test.cc
index 1548ca4..36c6ddb 100644
--- a/test/symbol-test.cc
+++ b/test/symbol-test.cc
@@ -4,7 +4,8 @@
int main() {
- (void) notmuch_database_open("fakedb", NOTMUCH_DATABASE_MODE_READ_ONLY);
+ (void) notmuch_database_open (notmuch_mailstore_get_by_name ("maildir"),
+ "fakedb", NOTMUCH_DATABASE_MODE_READ_ONLY);
try {
(void) new Xapian::WritableDatabase("./nonexistant", Xapian::DB_OPEN);
--
1.7.5.4
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [RFC PATCH 03/13] Introduce mailstore in the python bindings
2012-02-15 22:01 [RFC PATCH 00/13] Modular message store code Ethan Glasser-Camp
2012-02-15 22:01 ` [RFC PATCH 01/13] Create configuration paramater database.type Ethan Glasser-Camp
2012-02-15 22:01 ` [RFC PATCH 02/13] Add the concept of a mailstore in its absolute minimal sense Ethan Glasser-Camp
@ 2012-02-15 22:01 ` Ethan Glasser-Camp
2012-02-15 22:01 ` [RFC PATCH 04/13] Replace remaining places where fopen occurs Ethan Glasser-Camp
` (11 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Ethan Glasser-Camp @ 2012-02-15 22:01 UTC (permalink / raw)
To: notmuch; +Cc: Ethan Glasser-Camp
From: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
---
bindings/python/notmuch/database.py | 31 +++++++++++++++++---------
bindings/python/notmuch/globals.py | 3 ++
bindings/python/notmuch/mailstore.py | 38 ++++++++++++++++++++++++++++++++++
3 files changed, 61 insertions(+), 11 deletions(-)
create mode 100644 bindings/python/notmuch/mailstore.py
diff --git a/bindings/python/notmuch/database.py b/bindings/python/notmuch/database.py
index 36b65ec..638ade3 100644
--- a/bindings/python/notmuch/database.py
+++ b/bindings/python/notmuch/database.py
@@ -23,7 +23,9 @@ from ctypes import c_char_p, c_void_p, c_uint, c_long, byref, POINTER
from notmuch.globals import (nmlib, STATUS, NotmuchError, NotInitializedError,
NullPointerError, Enum, _str,
NotmuchDatabaseP, NotmuchDirectoryP, NotmuchMessageP, NotmuchTagsP,
- NotmuchQueryP, NotmuchMessagesP, NotmuchThreadsP, NotmuchFilenamesP)
+ NotmuchQueryP, NotmuchMessagesP, NotmuchThreadsP, NotmuchFilenamesP,
+ NotmuchMailstoreP)
+from notmuch.mailstore import Mailstore
from notmuch.thread import Threads
from notmuch.message import Messages, Message
from notmuch.tag import Tags
@@ -78,7 +80,7 @@ class Database(object):
"""notmuch_database_open"""
_open = nmlib.notmuch_database_open
- _open.argtypes = [c_char_p, c_uint]
+ _open.argtypes = [NotmuchMailstoreP, c_char_p, c_uint]
_open.restype = NotmuchDatabaseP
"""notmuch_database_upgrade"""
@@ -105,11 +107,13 @@ class Database(object):
"""notmuch_database_create"""
_create = nmlib.notmuch_database_create
- _create.argtypes = [c_char_p]
+ _create.argtypes = [NotmuchMailstoreP, c_char_p]
_create.restype = NotmuchDatabaseP
- def __init__(self, path=None, create=False, mode=0):
- """If *path* is `None`, we will try to read a users notmuch
+ def __init__(self, mailstore=None, path=None, create=False, mode=0):
+ """If *mailstore* is `None`, we will just use 'maildir'.
+
+ If *path* is `None`, we will try to read a users notmuch
configuration and use his configured database. The location of the
configuration file can be specified through the environment variable
*NOTMUCH_CONFIG*, falling back to the default `~/.notmuch-config`.
@@ -130,6 +134,9 @@ class Database(object):
failure.
"""
self._db = None
+ if mailstore == None:
+ mailstore = Mailstore('maildir')
+
if path is None:
# no path specified. use a user's default database
if Database._std_db_path is None:
@@ -138,16 +145,16 @@ class Database(object):
path = Database._std_db_path
if create == False:
- self.open(path, mode)
+ self.open(mailstore, path, mode)
else:
- self.create(path)
+ self.create(mailstore, path)
def _assert_db_is_initialized(self):
"""Raises :exc:`NotInitializedError` if self._db is `None`"""
if self._db is None:
raise NotInitializedError()
- def create(self, path):
+ def create(self, mailstore, path):
"""Creates a new notmuch database
This function is used by __init__() and usually does not need
@@ -167,14 +174,15 @@ class Database(object):
raise NotmuchError(message="Cannot create db, this Database() "
"already has an open one.")
- res = Database._create(_str(path), Database.MODE.READ_WRITE)
+ mailstore = mailstore._mailstore
+ res = Database._create(mailstore, _str(path), Database.MODE.READ_WRITE)
if not res:
raise NotmuchError(
message="Could not create the specified database")
self._db = res
- def open(self, path, mode=0):
+ def open(self, mailstore, path, mode=0):
"""Opens an existing database
This function is used by __init__() and usually does not need
@@ -187,7 +195,8 @@ class Database(object):
:exception: Raises :exc:`NotmuchError` in case of any failure
(possibly after printing an error message on stderr).
"""
- res = Database._open(_str(path), mode)
+ mailstore = mailstore._mailstore # unwrap mailstore
+ res = Database._open(mailstore, _str(path), mode)
if not res:
raise NotmuchError(message="Could not open the specified database")
diff --git a/bindings/python/notmuch/globals.py b/bindings/python/notmuch/globals.py
index 4138460..5f01dce 100644
--- a/bindings/python/notmuch/globals.py
+++ b/bindings/python/notmuch/globals.py
@@ -228,6 +228,9 @@ class NotmuchDatabaseS(Structure):
pass
NotmuchDatabaseP = POINTER(NotmuchDatabaseS)
+class NotmuchMailstoreS(Structure):
+ pass
+NotmuchMailstoreP = POINTER(NotmuchMailstoreS)
class NotmuchQueryS(Structure):
pass
diff --git a/bindings/python/notmuch/mailstore.py b/bindings/python/notmuch/mailstore.py
new file mode 100644
index 0000000..315ea70
--- /dev/null
+++ b/bindings/python/notmuch/mailstore.py
@@ -0,0 +1,38 @@
+"""
+This file is part of notmuch.
+
+Notmuch is free software: you can redistribute it and/or modify it
+under the terms of the GNU General Public License as published by the
+Free Software Foundation, either version 3 of the License, or (at your
+option) any later version.
+
+Notmuch is distributed in the hope that it will be useful, but WITHOUT
+ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
+for more details.
+
+You should have received a copy of the GNU General Public License
+along with notmuch. If not, see <http://www.gnu.org/licenses/>.
+
+Copyright 2012 Ethan Glasser-Camp <ethan@betacantrips.com>'
+"""
+
+# This is all kind of cargo-culted from database.py. I hope someone
+# else takes a good look at this!
+
+import os
+from ctypes import c_char_p, c_void_p, c_uint, c_long, byref, POINTER
+from notmuch.globals import (nmlib, STATUS, NotmuchError, NotInitializedError,
+ NullPointerError, Enum, _str,
+ NotmuchDatabaseP, NotmuchDirectoryP, NotmuchMessageP, NotmuchTagsP,
+ NotmuchQueryP, NotmuchMessagesP, NotmuchThreadsP, NotmuchFilenamesP,
+ NotmuchMailstoreP,)
+
+class Mailstore(object):
+ """The :class:`Mailstore` represents "where the mail lives"."""
+ _get_by_name = nmlib.notmuch_mailstore_get_by_name
+ _get_by_name.argtypes = [c_char_p]
+ _get_by_name.restype = NotmuchMailstoreP
+
+ def __init__(self, type=None, path=None):
+ self._mailstore = self._get_by_name(type)
--
1.7.5.4
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [RFC PATCH 04/13] Replace remaining places where fopen occurs
2012-02-15 22:01 [RFC PATCH 00/13] Modular message store code Ethan Glasser-Camp
` (2 preceding siblings ...)
2012-02-15 22:01 ` [RFC PATCH 03/13] Introduce mailstore in the python bindings Ethan Glasser-Camp
@ 2012-02-15 22:01 ` Ethan Glasser-Camp
2012-02-15 22:01 ` [RFC PATCH 05/13] notmuch_database_add_message calculates sha1 of messages using mailstore Ethan Glasser-Camp
` (10 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Ethan Glasser-Camp @ 2012-02-15 22:01 UTC (permalink / raw)
To: notmuch; +Cc: Ethan Glasser-Camp
From: Ethan Glasser-Camp <ethan@betacantrips.com>
Because mail might no longer be on disk, other uses of fopen(2) need
to be replaced with calls to notmuch_mailstore_open. This isn't all of
them, but these are the ones that involve touching the API in a lot of
different places.
This commit updates mime_node_open, show_message_body, and a couple
random calls in the commands notmuch show, notmuch reply,
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
---
mime-node.c | 3 ++-
notmuch-client.h | 2 ++
notmuch-reply.c | 2 +-
notmuch-show.c | 9 +++++----
show-message.c | 3 ++-
5 files changed, 12 insertions(+), 7 deletions(-)
diff --git a/mime-node.c b/mime-node.c
index d6b4506..856fc3b 100644
--- a/mime-node.c
+++ b/mime-node.c
@@ -61,6 +61,7 @@ _mime_node_context_free (mime_node_context_t *res)
notmuch_status_t
mime_node_open (const void *ctx, notmuch_message_t *message,
+ notmuch_mailstore_t *mailstore,
#ifdef GMIME_ATLEAST_26
GMimeCryptoContext *cryptoctx,
#else
@@ -89,7 +90,7 @@ mime_node_open (const void *ctx, notmuch_message_t *message,
}
talloc_set_destructor (mctx, _mime_node_context_free);
- mctx->file = fopen (filename, "r");
+ mctx->file = notmuch_mailstore_open (mailstore, filename);
if (! mctx->file) {
fprintf (stderr, "Error opening %s: %s\n", filename, strerror (errno));
status = NOTMUCH_STATUS_FILE_ERROR;
diff --git a/notmuch-client.h b/notmuch-client.h
index c1c30a2..405aad7 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -188,6 +188,7 @@ query_string_from_args (void *ctx, int argc, char *argv[]);
notmuch_status_t
show_message_body (notmuch_message_t *message,
+ notmuch_mailstore_t *mailstore,
const notmuch_show_format_t *format,
notmuch_show_params_t *params);
@@ -372,6 +373,7 @@ typedef struct mime_node {
*/
notmuch_status_t
mime_node_open (const void *ctx, notmuch_message_t *message,
+ notmuch_mailstore_t *mailstore,
#ifdef GMIME_ATLEAST_26
GMimeCryptoContext *cryptoctx,
#else
diff --git a/notmuch-reply.c b/notmuch-reply.c
index cb1dd6e..523e2d0 100644
--- a/notmuch-reply.c
+++ b/notmuch-reply.c
@@ -587,7 +587,7 @@ notmuch_reply_format_default(void *ctx,
notmuch_message_get_header (mailstore, message, "date"),
notmuch_message_get_header (mailstore, message, "from"));
- show_message_body (message, format, params);
+ show_message_body (message, mailstore, format, params);
notmuch_message_destroy (message);
}
diff --git a/notmuch-show.c b/notmuch-show.c
index 81d4cf0..0d2a246 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -285,7 +285,7 @@ format_message_mbox (const void *ctx,
ssize_t line_len;
filename = notmuch_message_get_filename (message);
- file = fopen (filename, "r");
+ file = notmuch_mailstore_open (mailstore, filename);
if (file == NULL) {
fprintf (stderr, "Failed to open %s: %s\n",
filename, strerror (errno));
@@ -830,7 +830,8 @@ show_message (void *ctx,
void *local = talloc_new (ctx);
mime_node_t *root, *part;
- if (mime_node_open (local, message, params->cryptoctx, params->decrypt,
+ if (mime_node_open (local, message, mailstore,
+ params->cryptoctx, params->decrypt,
&root) == NOTMUCH_STATUS_SUCCESS &&
(part = mime_node_seek_dfs (root, (params->part < 0 ?
0 : params->part))))
@@ -853,7 +854,7 @@ show_message (void *ctx,
}
if (format->part_content)
- show_message_body (message, format, params);
+ show_message_body (message, mailstore, format, params);
if (params->part <= 0) {
fputs (format->body_end, stdout);
@@ -955,7 +956,7 @@ do_show_single (void *ctx,
return 1;
}
- file = fopen (filename, "r");
+ file = notmuch_mailstore_open (mailstore, filename);
if (file == NULL) {
fprintf (stderr, "Error: Cannot open file %s: %s\n", filename, strerror (errno));
return 1;
diff --git a/show-message.c b/show-message.c
index 83ecf81..aed9d3e 100644
--- a/show-message.c
+++ b/show-message.c
@@ -80,6 +80,7 @@ show_message_part (mime_node_t *node,
notmuch_status_t
show_message_body (notmuch_message_t *message,
+ notmuch_mailstore_t *mailstore,
const notmuch_show_format_t *format,
notmuch_show_params_t *params)
{
@@ -87,7 +88,7 @@ show_message_body (notmuch_message_t *message,
show_message_state_t state;
mime_node_t *root, *part;
- ret = mime_node_open (NULL, message, params->cryptoctx, params->decrypt,
+ ret = mime_node_open (NULL, message, mailstore, params->cryptoctx, params->decrypt,
&root);
if (ret)
return ret;
--
1.7.5.4
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [RFC PATCH 05/13] notmuch_database_add_message calculates sha1 of messages using mailstore
2012-02-15 22:01 [RFC PATCH 00/13] Modular message store code Ethan Glasser-Camp
` (3 preceding siblings ...)
2012-02-15 22:01 ` [RFC PATCH 04/13] Replace remaining places where fopen occurs Ethan Glasser-Camp
@ 2012-02-15 22:01 ` Ethan Glasser-Camp
2012-02-15 22:01 ` [RFC PATCH 06/13] Pass mailstore to _notmuch_message_index_file Ethan Glasser-Camp
` (9 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Ethan Glasser-Camp @ 2012-02-15 22:01 UTC (permalink / raw)
To: notmuch; +Cc: Ethan Glasser-Camp
From: Ethan Glasser-Camp <ethan@betacantrips.com>
Previously, notmuch_database_add_message used the notmuch_sha1_of_file
function, which accesses a mail file directly. Create a new function
called notmuch_sha1_of_message which uses a mailstore to access the
file, and use that instead.
Also, as a drive-by cleanup, use the named constant BLOCK_SIZE instead
of the integer literal 4096.
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
---
lib/database.cc | 2 +-
lib/notmuch-private.h | 3 ++
lib/sha1.c | 52 ++++++++++++++++++++++++++++++++++++------------
3 files changed, 43 insertions(+), 14 deletions(-)
diff --git a/lib/database.cc b/lib/database.cc
index e3c8095..ff44e76 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1700,7 +1700,7 @@ notmuch_database_add_message (notmuch_database_t *notmuch,
if (message_id == NULL ) {
/* No message-id at all, let's generate one by taking a
* hash over the file's contents. */
- char *sha1 = notmuch_sha1_of_file (filename);
+ char *sha1 = notmuch_sha1_of_message (notmuch->mailstore, filename);
/* If that failed too, something is really wrong. Give up. */
if (sha1 == NULL) {
diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index 0f01437..2589928 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -454,6 +454,9 @@ notmuch_sha1_of_string (const char *str);
char *
notmuch_sha1_of_file (const char *filename);
+char *
+notmuch_sha1_of_message (notmuch_mailstore_t *mailstore, const char *filename);
+
/* string-list.c */
typedef struct _notmuch_string_node {
diff --git a/lib/sha1.c b/lib/sha1.c
index cc48108..ea25999 100644
--- a/lib/sha1.c
+++ b/lib/sha1.c
@@ -64,19 +64,11 @@ notmuch_sha1_of_string (const char *str)
return _hex_of_sha1_digest (digest);
}
-/* Create a hexadecimal string version of the SHA-1 digest of the
- * contents of the named file.
- *
- * This function returns a newly allocated string which the caller
- * should free() when finished.
- *
- * If any error occurs while reading the file, (permission denied,
- * file not found, etc.), this function returns NULL.
+/* Internal function to feed the contents of a FILE * to the sha1 functions.
*/
-char *
-notmuch_sha1_of_file (const char *filename)
+static char *
+_notmuch_sha1_of_filep (FILE *file)
{
- FILE *file;
#define BLOCK_SIZE 4096
unsigned char block[BLOCK_SIZE];
size_t bytes_read;
@@ -84,14 +76,13 @@ notmuch_sha1_of_file (const char *filename)
unsigned char digest[SHA1_DIGEST_SIZE];
char *result;
- file = fopen (filename, "r");
if (file == NULL)
return NULL;
sha1_begin (&sha1);
while (1) {
- bytes_read = fread (block, 1, 4096, file);
+ bytes_read = fread (block, 1, BLOCK_SIZE, file);
if (bytes_read == 0) {
if (feof (file)) {
break;
@@ -113,3 +104,38 @@ notmuch_sha1_of_file (const char *filename)
return result;
}
+/* Create a hexadecimal string version of the SHA-1 digest of the
+ * contents of the named file.
+ *
+ * This function returns a newly allocated string which the caller
+ * should free() when finished.
+ *
+ * If any error occurs while reading the file, (permission denied,
+ * file not found, etc.), this function returns NULL.
+ */
+char *
+notmuch_sha1_of_file (const char *filename)
+{
+ FILE *file;
+
+ file = fopen (filename, "r");
+ return _notmuch_sha1_of_filep (file);
+}
+
+/* Create a hexadecimal string version of the SHA-1 digest of the
+ * contents of the named message, which is accessed via a mailstore.
+ *
+ * This function returns a newly allocated string which the caller
+ * should free() when finished.
+ *
+ * If any error occurs while reading the file, (permission denied,
+ * file not found, etc.), this function returns NULL.
+ */
+char *
+notmuch_sha1_of_message (notmuch_mailstore_t *mailstore, const char *filename)
+{
+ FILE *file;
+
+ file = notmuch_mailstore_open (mailstore, filename);
+ return _notmuch_sha1_of_filep (file);
+}
--
1.7.5.4
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [RFC PATCH 06/13] Pass mailstore to _notmuch_message_index_file
2012-02-15 22:01 [RFC PATCH 00/13] Modular message store code Ethan Glasser-Camp
` (4 preceding siblings ...)
2012-02-15 22:01 ` [RFC PATCH 05/13] notmuch_database_add_message calculates sha1 of messages using mailstore Ethan Glasser-Camp
@ 2012-02-15 22:01 ` Ethan Glasser-Camp
2012-02-15 22:02 ` [RFC PATCH 07/13] notmuch-new: pull out useful bits of add_files_recursive Ethan Glasser-Camp
` (8 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Ethan Glasser-Camp @ 2012-02-15 22:01 UTC (permalink / raw)
To: notmuch; +Cc: Ethan Glasser-Camp
From: Ethan Glasser-Camp <ethan@betacantrips.com>
This is the last place where fopen(2) was used and had to be replaced.
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
---
lib/database.cc | 2 +-
lib/index.cc | 5 +++--
lib/notmuch-private.h | 3 ++-
3 files changed, 6 insertions(+), 4 deletions(-)
diff --git a/lib/database.cc b/lib/database.cc
index ff44e76..0ed4412 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1743,7 +1743,7 @@ notmuch_database_add_message (notmuch_database_t *notmuch,
date = notmuch_message_file_get_header (message_file, "date");
_notmuch_message_set_header_values (message, date, from, subject);
- _notmuch_message_index_file (message, filename);
+ _notmuch_message_index_file (notmuch->mailstore, message, filename);
} else {
ret = NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID;
}
diff --git a/lib/index.cc b/lib/index.cc
index d8f8b2b..54b8f73 100644
--- a/lib/index.cc
+++ b/lib/index.cc
@@ -409,7 +409,8 @@ _index_mime_part (notmuch_message_t *message,
}
notmuch_status_t
-_notmuch_message_index_file (notmuch_message_t *message,
+_notmuch_message_index_file (notmuch_mailstore_t *mailstore,
+ notmuch_message_t *message,
const char *filename)
{
GMimeStream *stream = NULL;
@@ -426,7 +427,7 @@ _notmuch_message_index_file (notmuch_message_t *message,
initialized = 1;
}
- file = fopen (filename, "r");
+ file = notmuch_mailstore_open (mailstore, filename);
if (! file) {
fprintf (stderr, "Error opening %s: %s\n", filename, strerror (errno));
ret = NOTMUCH_STATUS_FILE_ERROR;
diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index 2589928..d93891e 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -313,7 +313,8 @@ notmuch_message_get_author (notmuch_message_t *message);
/* index.cc */
notmuch_status_t
-_notmuch_message_index_file (notmuch_message_t *message,
+_notmuch_message_index_file (notmuch_mailstore_t *mailstore,
+ notmuch_message_t *message,
const char *filename);
/* message-file.c */
--
1.7.5.4
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [RFC PATCH 07/13] notmuch-new: pull out useful bits of add_files_recursive
2012-02-15 22:01 [RFC PATCH 00/13] Modular message store code Ethan Glasser-Camp
` (5 preceding siblings ...)
2012-02-15 22:01 ` [RFC PATCH 06/13] Pass mailstore to _notmuch_message_index_file Ethan Glasser-Camp
@ 2012-02-15 22:02 ` Ethan Glasser-Camp
2012-02-15 22:02 ` [RFC PATCH 08/13] count_files and add_files shall be generic Ethan Glasser-Camp
` (7 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Ethan Glasser-Camp @ 2012-02-15 22:02 UTC (permalink / raw)
To: notmuch; +Cc: Ethan Glasser-Camp
From: Ethan Glasser-Camp <ethan@betacantrips.com>
This is part of notmuch-new refactor phase 1: make add_files stuff
safe for other backends. add_files_recursive is essentially a
maildir-crawling function that periodically adds files to the database
or adds filenames to remove_files or remove_directory lists. I don't
see an easy way to adapt add_files_recursive for other backends who
might not have concepts of directories with other directories inside
of them, so instead just provide an add_files method for each backend.
This patch pulls some bits out of add_files_recursive which will be
useful for other backends: two reporting functions
_report_before_adding_file and _report_added_file, as well as
_add_message, which actually does the message adding.
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
---
notmuch-new.c | 193 +++++++++++++++++++++++++++++++++++----------------------
1 files changed, 120 insertions(+), 73 deletions(-)
diff --git a/notmuch-new.c b/notmuch-new.c
index 355dde4..dbdfbb6 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -181,6 +181,123 @@ _entries_resemble_maildir (struct dirent **entries, int count)
return 0;
}
+/* Progress-reporting function.
+ *
+ * Can be used by any mailstore-crawling function that wants to alert
+ * users what message it's about to add. Subsequent errors will be due
+ * to this message ;)
+ */
+static void
+_report_before_adding_file (add_files_state_t *state, const char *filename)
+{
+ state->processed_files++;
+
+ if (state->verbose) {
+ if (state->output_is_a_tty)
+ printf("\r\033[K");
+
+ printf ("%i/%i: %s",
+ state->processed_files,
+ state->total_files,
+ filename);
+
+ putchar((state->output_is_a_tty) ? '\r' : '\n');
+ fflush (stdout);
+ }
+}
+
+/* Progress-reporting function.
+ *
+ * Call this to respond to the signal handler for SIGALRM.
+ */
+static void
+_report_added_file (add_files_state_t *state)
+{
+ if (do_print_progress) {
+ do_print_progress = 0;
+ generic_print_progress ("Processed", "files", state->tv_start,
+ state->processed_files, state->total_files);
+ }
+}
+
+
+/* Atomically handles adding a message to the database.
+ *
+ * Should be used by any mailstore-crawling function that finds a new
+ * message to add.
+ */
+static notmuch_status_t
+_add_message (add_files_state_t *state, notmuch_database_t *notmuch,
+ const char *filename)
+{
+ notmuch_status_t status, ret = NOTMUCH_STATUS_SUCCESS;
+ notmuch_message_t *message;
+ const char **tag;
+
+ status = notmuch_database_begin_atomic (notmuch);
+ if (status) {
+ ret = status;
+ goto DONE;
+ }
+
+ status = notmuch_database_add_message (notmuch, filename, &message);
+
+ switch (status) {
+ /* success */
+ case NOTMUCH_STATUS_SUCCESS:
+ state->added_messages++;
+ notmuch_message_freeze (message);
+ for (tag=state->new_tags; *tag != NULL; tag++)
+ notmuch_message_add_tag (message, *tag);
+ if (state->synchronize_flags == TRUE)
+ notmuch_message_maildir_flags_to_tags (message);
+ notmuch_message_thaw (message);
+ break;
+ /* Non-fatal issues (go on to next file) */
+ case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
+ if (state->synchronize_flags == TRUE)
+ notmuch_message_maildir_flags_to_tags (message);
+ break;
+ case NOTMUCH_STATUS_FILE_NOT_EMAIL:
+ fprintf (stderr, "Note: Ignoring non-mail file: %s\n",
+ filename);
+ break;
+ /* Fatal issues. Don't process anymore. */
+ case NOTMUCH_STATUS_READ_ONLY_DATABASE:
+ case NOTMUCH_STATUS_XAPIAN_EXCEPTION:
+ case NOTMUCH_STATUS_OUT_OF_MEMORY:
+ fprintf (stderr, "Error: %s. Halting processing.\n",
+ notmuch_status_to_string (status));
+ ret = status;
+ goto DONE;
+ default:
+ case NOTMUCH_STATUS_FILE_ERROR:
+ case NOTMUCH_STATUS_NULL_POINTER:
+ case NOTMUCH_STATUS_TAG_TOO_LONG:
+ case NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW:
+ case NOTMUCH_STATUS_UNBALANCED_ATOMIC:
+ case NOTMUCH_STATUS_LAST_STATUS:
+ INTERNAL_ERROR ("add_message returned unexpected value: %d", status);
+ ret = status;
+ goto DONE;
+ }
+
+ status = notmuch_database_end_atomic (notmuch);
+ if (status) {
+ ret = status;
+ goto DONE;
+ }
+
+ DONE:
+ if (message) {
+ notmuch_message_destroy (message);
+ message = NULL;
+ }
+
+ return ret;
+}
+
+
/* Examine 'path' recursively as follows:
*
* o Ask the filesystem for the mtime of 'path' (fs_mtime)
@@ -232,7 +349,6 @@ add_files_recursive (notmuch_database_t *notmuch,
char *next = NULL;
time_t fs_mtime, db_mtime;
notmuch_status_t status, ret = NOTMUCH_STATUS_SUCCESS;
- notmuch_message_t *message = NULL;
struct dirent **fs_entries = NULL;
int i, num_fs_entries;
notmuch_directory_t *directory;
@@ -241,7 +357,6 @@ add_files_recursive (notmuch_database_t *notmuch,
time_t stat_time;
struct stat st;
notmuch_bool_t is_maildir, new_directory;
- const char **tag;
if (stat (path, &st)) {
fprintf (stderr, "Error reading directory %s: %s\n",
@@ -439,83 +554,15 @@ add_files_recursive (notmuch_database_t *notmuch,
* in the database, so add it. */
next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
- state->processed_files++;
-
- if (state->verbose) {
- if (state->output_is_a_tty)
- printf("\r\033[K");
+ _report_before_adding_file (state, next);
- printf ("%i/%i: %s",
- state->processed_files,
- state->total_files,
- next);
-
- putchar((state->output_is_a_tty) ? '\r' : '\n');
- fflush (stdout);
- }
-
- status = notmuch_database_begin_atomic (notmuch);
+ status = _add_message (state, notmuch, next);
if (status) {
ret = status;
goto DONE;
}
- status = notmuch_database_add_message (notmuch, next, &message);
- switch (status) {
- /* success */
- case NOTMUCH_STATUS_SUCCESS:
- state->added_messages++;
- notmuch_message_freeze (message);
- for (tag=state->new_tags; *tag != NULL; tag++)
- notmuch_message_add_tag (message, *tag);
- if (state->synchronize_flags == TRUE)
- notmuch_message_maildir_flags_to_tags (message);
- notmuch_message_thaw (message);
- break;
- /* Non-fatal issues (go on to next file) */
- case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
- if (state->synchronize_flags == TRUE)
- notmuch_message_maildir_flags_to_tags (message);
- break;
- case NOTMUCH_STATUS_FILE_NOT_EMAIL:
- fprintf (stderr, "Note: Ignoring non-mail file: %s\n",
- next);
- break;
- /* Fatal issues. Don't process anymore. */
- case NOTMUCH_STATUS_READ_ONLY_DATABASE:
- case NOTMUCH_STATUS_XAPIAN_EXCEPTION:
- case NOTMUCH_STATUS_OUT_OF_MEMORY:
- fprintf (stderr, "Error: %s. Halting processing.\n",
- notmuch_status_to_string (status));
- ret = status;
- goto DONE;
- default:
- case NOTMUCH_STATUS_FILE_ERROR:
- case NOTMUCH_STATUS_NULL_POINTER:
- case NOTMUCH_STATUS_TAG_TOO_LONG:
- case NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW:
- case NOTMUCH_STATUS_UNBALANCED_ATOMIC:
- case NOTMUCH_STATUS_LAST_STATUS:
- INTERNAL_ERROR ("add_message returned unexpected value: %d", status);
- goto DONE;
- }
-
- status = notmuch_database_end_atomic (notmuch);
- if (status) {
- ret = status;
- goto DONE;
- }
-
- if (message) {
- notmuch_message_destroy (message);
- message = NULL;
- }
-
- if (do_print_progress) {
- do_print_progress = 0;
- generic_print_progress ("Processed", "files", state->tv_start,
- state->processed_files, state->total_files);
- }
+ _report_added_file (state);
talloc_free (next);
next = NULL;
--
1.7.5.4
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [RFC PATCH 08/13] count_files and add_files shall be generic
2012-02-15 22:01 [RFC PATCH 00/13] Modular message store code Ethan Glasser-Camp
` (6 preceding siblings ...)
2012-02-15 22:02 ` [RFC PATCH 07/13] notmuch-new: pull out useful bits of add_files_recursive Ethan Glasser-Camp
@ 2012-02-15 22:02 ` Ethan Glasser-Camp
2012-02-15 22:02 ` [RFC PATCH 09/13] Mailstore also provides a "rename" function Ethan Glasser-Camp
` (6 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Ethan Glasser-Camp @ 2012-02-15 22:02 UTC (permalink / raw)
To: notmuch; +Cc: Ethan Glasser-Camp
From: Ethan Glasser-Camp <ethan@betacantrips.com>
Rename current count_files and add_files to maildir_count_files and
maildir_add_files. This allows the possibility, at least, of having
other backends.
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
---
notmuch-new.c | 62 +++++++++++++++++++++++++++++++++++++++++++++-----------
1 files changed, 50 insertions(+), 12 deletions(-)
diff --git a/notmuch-new.c b/notmuch-new.c
index dbdfbb6..d30fba1 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -340,9 +340,9 @@ _add_message (add_files_state_t *state, notmuch_database_t *notmuch,
* if fs_mtime isn't the current wall-clock time.
*/
static notmuch_status_t
-add_files_recursive (notmuch_database_t *notmuch,
- const char *path,
- add_files_state_t *state)
+maildir_add_files_recursive (notmuch_database_t *notmuch,
+ const char *path,
+ add_files_state_t *state)
{
DIR *dir = NULL;
struct dirent *entry = NULL;
@@ -449,7 +449,7 @@ add_files_recursive (notmuch_database_t *notmuch,
}
next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
- status = add_files_recursive (notmuch, next, state);
+ status = maildir_add_files_recursive (notmuch, next, state);
if (status && ret == NOTMUCH_STATUS_SUCCESS)
ret = status;
talloc_free (next);
@@ -663,13 +663,34 @@ stop_progress_printing_timer (void)
sigaction (SIGALRM, &action, NULL);
}
+static notmuch_status_t
+maildir_add_files (notmuch_database_t *notmuch,
+ const char *path,
+ add_files_state_t *state);
+
+/* Dispatch function to call the correct mailstore_add_files
+ * function. */
+static notmuch_status_t
+add_files (notmuch_database_t *notmuch, notmuch_config_t *config,
+ add_files_state_t *state)
+{
+ const char *path = notmuch_config_get_database_path (config);
+ if (strcmp (notmuch_config_get_database_type (config), "maildir") == 0)
+ return maildir_add_files (notmuch, path, state);
+
+ /* Default case */
+ fprintf (stderr, "Could not add files for mailstore %s: unknown mailstore\n",
+ notmuch_config_get_database_type (config));
+ /* FIXME: "invalid argument" error code would be nice */
+ return NOTMUCH_STATUS_FILE_ERROR;
+}
/* This is the top-level entry point for add_files. It does a couple
* of error checks and then calls into the recursive function. */
static notmuch_status_t
-add_files (notmuch_database_t *notmuch,
- const char *path,
- add_files_state_t *state)
+maildir_add_files (notmuch_database_t *notmuch,
+ const char *path,
+ add_files_state_t *state)
{
notmuch_status_t status;
struct stat st;
@@ -685,11 +706,28 @@ add_files (notmuch_database_t *notmuch,
return NOTMUCH_STATUS_FILE_ERROR;
}
- status = add_files_recursive (notmuch, path, state);
+ status = maildir_add_files_recursive (notmuch, path, state);
return status;
}
+static void
+maildir_count_files (const char *path, int *count);
+
+/* Dispatch function to call the correct mailstore_count_files
+ * function. N.B. This function may get refactored into add_files! */
+static void
+count_files (unused (notmuch_database_t *notmuch), notmuch_config_t *config, int *count)
+{
+ if (strcmp (notmuch_config_get_database_type (config), "maildir") == 0) {
+ maildir_count_files (notmuch_config_get_database_path (config), count);
+ return;
+ }
+
+ /* Eh, screw it, this function shouldn't even exist */
+ *count = 0;
+}
+
/* XXX: This should be merged with the add_files function since it
* shares a lot of logic with it. */
/* Recursively count all regular files in path and all sub-directories
@@ -697,7 +735,7 @@ add_files (notmuch_database_t *notmuch,
* initialized to zero by the top-level caller before calling
* count_files). */
static void
-count_files (const char *path, int *count)
+maildir_count_files (const char *path, int *count)
{
struct dirent *entry = NULL;
char *next;
@@ -746,7 +784,7 @@ count_files (const char *path, int *count)
fflush (stdout);
}
} else if (S_ISDIR (st.st_mode)) {
- count_files (next, count);
+ maildir_count_files (next, count);
}
free (next);
@@ -905,7 +943,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
int count;
count = 0;
- count_files (db_path, &count);
+ count_files (notmuch, config, &count);
if (interrupted)
return 1;
@@ -962,7 +1000,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
timer_is_active = TRUE;
}
- ret = add_files (notmuch, db_path, &add_files_state);
+ ret = add_files (notmuch, config, &add_files_state);
gettimeofday (&tv_start, NULL);
for (f = add_files_state.removed_files->head; f && !interrupted; f = f->next) {
--
1.7.5.4
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [RFC PATCH 09/13] Mailstore also provides a "rename" function
2012-02-15 22:01 [RFC PATCH 00/13] Modular message store code Ethan Glasser-Camp
` (7 preceding siblings ...)
2012-02-15 22:02 ` [RFC PATCH 08/13] count_files and add_files shall be generic Ethan Glasser-Camp
@ 2012-02-15 22:02 ` Ethan Glasser-Camp
2012-02-15 22:02 ` [RFC PATCH 10/13] Introduce concept of mailstore "constructor" Ethan Glasser-Camp
` (5 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Ethan Glasser-Camp @ 2012-02-15 22:02 UTC (permalink / raw)
To: notmuch; +Cc: Ethan Glasser-Camp
From: Ethan Glasser-Camp <ethan@betacantrips.com>
This is used only in notmuch_message_tags_to_maildir_flags, to update
a message's filename when its tags change. For mailstores where this
doesn't make sense, they can of course define rename to be a NOOP.
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
---
lib/mailstore.c | 21 ++++++++++++++++++++-
lib/message.cc | 5 +++--
lib/notmuch.h | 10 +++++++++-
notmuch-restore.c | 7 +++++--
notmuch-tag.c | 7 +++++--
5 files changed, 42 insertions(+), 8 deletions(-)
diff --git a/lib/mailstore.c b/lib/mailstore.c
index 290da70..2c6beab 100644
--- a/lib/mailstore.c
+++ b/lib/mailstore.c
@@ -22,6 +22,8 @@
typedef struct _notmuch_mailstore {
FILE *(*open) (struct _notmuch_mailstore *mailstore, const char *filename);
+ int (*rename) (struct _notmuch_mailstore *mailstore, const char *old_filename,
+ const char *new_filename);
} _notmuch_mailstore;
static FILE *
@@ -31,17 +33,27 @@ _maildir_open_function (unused (notmuch_mailstore_t *mailstore),
return fopen (filename, "r");
}
+static int
+_maildir_rename_function (unused (notmuch_mailstore_t *mailstore),
+ const char *old_filename, const char *new_filename)
+{
+ return rename (old_filename, new_filename);
+}
+
/* A mailstore is defined as:
*
* - A function used to "open" a mail message. This takes the
* "filename" for the file and should return a FILE *.
*
+ * - A function to "rename" a mail message, which is currently only
+ * used in tags_to_maildir_flags.
+ *
* - TODO: A way to scan for new messages?
*
* - TODO: A "constructor"?
*/
_notmuch_mailstore
-notmuch_mailstore_maildir = { _maildir_open_function };
+notmuch_mailstore_maildir = { _maildir_open_function, _maildir_rename_function };
_notmuch_mailstore *
notmuch_mailstore_get_by_name (const char *name)
@@ -57,3 +69,10 @@ notmuch_mailstore_open (notmuch_mailstore_t *mailstore, const char *filename)
{
return mailstore->open (mailstore, filename);
}
+
+int
+notmuch_mailstore_rename (notmuch_mailstore_t *mailstore, const char *old_filename,
+ const char *new_filename)
+{
+ return mailstore->rename (mailstore, old_filename, new_filename);
+}
diff --git a/lib/message.cc b/lib/message.cc
index 762a18f..cd81f6b 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -1291,7 +1291,8 @@ _new_maildir_filename (void *ctx,
}
notmuch_status_t
-notmuch_message_tags_to_maildir_flags (notmuch_message_t *message)
+notmuch_message_tags_to_maildir_flags (notmuch_mailstore_t *mailstore,
+ notmuch_message_t *message)
{
notmuch_filenames_t *filenames;
const char *filename;
@@ -1319,7 +1320,7 @@ notmuch_message_tags_to_maildir_flags (notmuch_message_t *message)
int err;
notmuch_status_t new_status;
- err = rename (filename, filename_new);
+ err = notmuch_mailstore_rename (mailstore, filename, filename_new);
if (err)
continue;
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 7ebe034..b6e66a9 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -423,6 +423,13 @@ notmuch_mailstore_get_by_name (const char *name);
FILE *
notmuch_mailstore_open (notmuch_mailstore_t *mailstore, const char *filename);
+/* Rename a file. This is used to update maildir tags and can safely
+ * be a NO-OP for non-filesystem mailstores.
+ */
+int
+notmuch_mailstore_rename (notmuch_mailstore_t *mailstore, const char *old_filename,
+ const char *new_filename);
+
/* Create a new query for 'database'.
*
* Here, 'database' should be an open database, (see
@@ -1095,7 +1102,8 @@ notmuch_message_maildir_flags_to_tags (notmuch_message_t *message);
* for synchronizing maildir flag changes back to tags.
*/
notmuch_status_t
-notmuch_message_tags_to_maildir_flags (notmuch_message_t *message);
+notmuch_message_tags_to_maildir_flags (notmuch_mailstore_t *mailstore,
+ notmuch_message_t *message);
/* Freeze the current state of 'message' within the database.
*
diff --git a/notmuch-restore.c b/notmuch-restore.c
index b382b7b..a0beab4 100644
--- a/notmuch-restore.c
+++ b/notmuch-restore.c
@@ -25,6 +25,7 @@ notmuch_restore_command (unused (void *ctx), int argc, char *argv[])
{
notmuch_config_t *config;
notmuch_database_t *notmuch;
+ notmuch_mailstore_t *mailstore;
notmuch_bool_t synchronize_flags;
notmuch_bool_t accumulate = FALSE;
char *input_file_name = NULL;
@@ -40,7 +41,9 @@ notmuch_restore_command (unused (void *ctx), int argc, char *argv[])
if (config == NULL)
return 1;
- notmuch = notmuch_database_open (notmuch_config_get_mailstore (config),
+ mailstore = notmuch_config_get_mailstore (config);
+
+ notmuch = notmuch_database_open (mailstore,
notmuch_config_get_database_path (config),
NOTMUCH_DATABASE_MODE_READ_WRITE);
if (notmuch == NULL)
@@ -170,7 +173,7 @@ notmuch_restore_command (unused (void *ctx), int argc, char *argv[])
notmuch_message_thaw (message);
if (synchronize_flags)
- notmuch_message_tags_to_maildir_flags (message);
+ notmuch_message_tags_to_maildir_flags (mailstore, message);
NEXT_LINE:
if (message)
diff --git a/notmuch-tag.c b/notmuch-tag.c
index 5e8d74a..0e0a7b4 100644
--- a/notmuch-tag.c
+++ b/notmuch-tag.c
@@ -122,6 +122,7 @@ notmuch_tag_command (void *ctx, int argc, char *argv[])
notmuch_query_t *query;
notmuch_messages_t *messages;
notmuch_message_t *message;
+ notmuch_mailstore_t *mailstore;
struct sigaction action;
notmuch_bool_t synchronize_flags;
int i;
@@ -187,7 +188,9 @@ notmuch_tag_command (void *ctx, int argc, char *argv[])
if (config == NULL)
return 1;
- notmuch = notmuch_database_open (notmuch_config_get_mailstore (config),
+ mailstore = notmuch_config_get_mailstore (config);
+
+ notmuch = notmuch_database_open (mailstore,
notmuch_config_get_database_path (config),
NOTMUCH_DATABASE_MODE_READ_WRITE);
if (notmuch == NULL)
@@ -222,7 +225,7 @@ notmuch_tag_command (void *ctx, int argc, char *argv[])
notmuch_message_thaw (message);
if (synchronize_flags)
- notmuch_message_tags_to_maildir_flags (message);
+ notmuch_message_tags_to_maildir_flags (mailstore, message);
notmuch_message_destroy (message);
}
--
1.7.5.4
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [RFC PATCH 10/13] Introduce concept of mailstore "constructor"
2012-02-15 22:01 [RFC PATCH 00/13] Modular message store code Ethan Glasser-Camp
` (8 preceding siblings ...)
2012-02-15 22:02 ` [RFC PATCH 09/13] Mailstore also provides a "rename" function Ethan Glasser-Camp
@ 2012-02-15 22:02 ` Ethan Glasser-Camp
2012-02-15 22:02 ` [RFC PATCH 11/13] Add a close function to mailstore Ethan Glasser-Camp
` (4 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Ethan Glasser-Camp @ 2012-02-15 22:02 UTC (permalink / raw)
To: notmuch; +Cc: Ethan Glasser-Camp
From: Ethan Glasser-Camp <ethan@betacantrips.com>
Right now this is a fancy no-op because maildir doesn't need any
special data, but getting the API right is good. A constructor can
fail, so return a notmuch_status_t.
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
---
lib/mailstore.c | 36 +++++++++++++++++++++++++++++++++---
lib/notmuch.h | 7 +++++++
notmuch-config.c | 8 +++++++-
3 files changed, 47 insertions(+), 4 deletions(-)
diff --git a/lib/mailstore.c b/lib/mailstore.c
index 2c6beab..b4d512d 100644
--- a/lib/mailstore.c
+++ b/lib/mailstore.c
@@ -17,15 +17,25 @@
*/
#include <stdio.h>
+#include <stdarg.h>
#include "notmuch-private.h"
typedef struct _notmuch_mailstore {
+ notmuch_status_t (*constructor) (void **data, va_list args);
FILE *(*open) (struct _notmuch_mailstore *mailstore, const char *filename);
int (*rename) (struct _notmuch_mailstore *mailstore, const char *old_filename,
const char *new_filename);
+ void *data;
} _notmuch_mailstore;
+static notmuch_status_t
+_maildir_constructor (void **data, unused (va_list ap))
+{
+ (*data) = NULL;
+ return NOTMUCH_STATUS_SUCCESS;
+}
+
static FILE *
_maildir_open_function (unused (notmuch_mailstore_t *mailstore),
const char *filename)
@@ -48,12 +58,18 @@ _maildir_rename_function (unused (notmuch_mailstore_t *mailstore),
* - A function to "rename" a mail message, which is currently only
* used in tags_to_maildir_flags.
*
- * - TODO: A way to scan for new messages?
+ * - A "constructor" that creates a mailstore object of the requisite
+ * type. Arguments are passed via va_args.
*
- * - TODO: A "constructor"?
+ * A mailstore also has a "data" field that can be used to store
+ * instance-specific information about this mailstore -- for example,
+ * a CouchDB URL or a path. FIXME: mailstores are all statically
+ * allocated, so maybe we shouldn't do this.
*/
_notmuch_mailstore
-notmuch_mailstore_maildir = { _maildir_open_function, _maildir_rename_function };
+notmuch_mailstore_maildir = { _maildir_constructor,
+ _maildir_open_function, _maildir_rename_function,
+ NULL };
_notmuch_mailstore *
notmuch_mailstore_get_by_name (const char *name)
@@ -76,3 +92,17 @@ notmuch_mailstore_rename (notmuch_mailstore_t *mailstore, const char *old_filena
{
return mailstore->rename (mailstore, old_filename, new_filename);
}
+
+notmuch_status_t
+notmuch_mailstore_construct (notmuch_mailstore_t *mailstore, ...)
+{
+ va_list va_args;
+ notmuch_status_t status;
+
+ va_start (va_args, mailstore);
+
+ status = mailstore->constructor (&mailstore->data, va_args);
+
+ va_end (va_args);
+ return status;
+}
diff --git a/lib/notmuch.h b/lib/notmuch.h
index b6e66a9..7f48507 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -430,6 +430,13 @@ int
notmuch_mailstore_rename (notmuch_mailstore_t *mailstore, const char *old_filename,
const char *new_filename);
+/* Initialize the mailstore.
+ *
+ * Arguments are dependent on the mailstore.
+ */
+notmuch_status_t
+notmuch_mailstore_construct (notmuch_mailstore_t *mailstore, ...);
+
/* Create a new query for 'database'.
*
* Here, 'database' should be an open database, (see
diff --git a/notmuch-config.c b/notmuch-config.c
index f611b26..99f872d 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -592,8 +592,14 @@ notmuch_config_get_mailstore (notmuch_config_t *config)
* When there are multiple mailstore types and "constructors" for
* them, this may have to be much more complicated.
*/
+ notmuch_status_t status;
const char *type = notmuch_config_get_database_type (config);
- return notmuch_mailstore_get_by_name (type);
+ notmuch_mailstore_t *mailstore = notmuch_mailstore_get_by_name (type);
+ status = notmuch_mailstore_construct (mailstore);
+ if (status != NOTMUCH_STATUS_SUCCESS) {
+ /* abort messily? */
+ }
+ return mailstore;
}
const char *
--
1.7.5.4
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [RFC PATCH 11/13] Add a close function to mailstore
2012-02-15 22:01 [RFC PATCH 00/13] Modular message store code Ethan Glasser-Camp
` (9 preceding siblings ...)
2012-02-15 22:02 ` [RFC PATCH 10/13] Introduce concept of mailstore "constructor" Ethan Glasser-Camp
@ 2012-02-15 22:02 ` Ethan Glasser-Camp
2012-02-15 22:02 ` [RFC PATCH 12/13] Close files using notmuch_mailstore_close instead of fclose Ethan Glasser-Camp
` (3 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Ethan Glasser-Camp @ 2012-02-15 22:02 UTC (permalink / raw)
To: notmuch; +Cc: Ethan Glasser-Camp
From: Ethan Glasser-Camp <ethan@betacantrips.com>
This is a useful way to signal to mailstores that the resources
associated with an existing FILE* are no longer being used and they
can be cleaned up. For maildir, of course, this is just a call to
fclose(), but for other mailstores this might involve freeing memory
or unlinking temporary files...
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
---
lib/mailstore.c | 17 ++++++++++++++++-
lib/notmuch.h | 5 +++++
2 files changed, 21 insertions(+), 1 deletions(-)
diff --git a/lib/mailstore.c b/lib/mailstore.c
index b4d512d..51c2710 100644
--- a/lib/mailstore.c
+++ b/lib/mailstore.c
@@ -24,6 +24,7 @@
typedef struct _notmuch_mailstore {
notmuch_status_t (*constructor) (void **data, va_list args);
FILE *(*open) (struct _notmuch_mailstore *mailstore, const char *filename);
+ int (*close) (struct _notmuch_mailstore *mailstore, FILE *);
int (*rename) (struct _notmuch_mailstore *mailstore, const char *old_filename,
const char *new_filename);
void *data;
@@ -44,6 +45,13 @@ _maildir_open_function (unused (notmuch_mailstore_t *mailstore),
}
static int
+_maildir_close_function (unused (notmuch_mailstore_t *mailstore),
+ FILE *file)
+{
+ return fclose (file);
+}
+
+static int
_maildir_rename_function (unused (notmuch_mailstore_t *mailstore),
const char *old_filename, const char *new_filename)
{
@@ -68,7 +76,8 @@ _maildir_rename_function (unused (notmuch_mailstore_t *mailstore),
*/
_notmuch_mailstore
notmuch_mailstore_maildir = { _maildir_constructor,
- _maildir_open_function, _maildir_rename_function,
+ _maildir_open_function, _maildir_close_function,
+ _maildir_rename_function,
NULL };
_notmuch_mailstore *
@@ -93,6 +102,12 @@ notmuch_mailstore_rename (notmuch_mailstore_t *mailstore, const char *old_filena
return mailstore->rename (mailstore, old_filename, new_filename);
}
+int
+notmuch_mailstore_close (notmuch_mailstore_t *mailstore, FILE *file)
+{
+ return mailstore->close (mailstore, file);
+}
+
notmuch_status_t
notmuch_mailstore_construct (notmuch_mailstore_t *mailstore, ...)
{
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 7f48507..d760d4f 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -423,6 +423,11 @@ notmuch_mailstore_get_by_name (const char *name);
FILE *
notmuch_mailstore_open (notmuch_mailstore_t *mailstore, const char *filename);
+/* Signal that a filename is no longer being used; get rid of its resources.
+ */
+int
+notmuch_mailstore_close (notmuch_mailstore_t *mailstore, FILE *file);
+
/* Rename a file. This is used to update maildir tags and can safely
* be a NO-OP for non-filesystem mailstores.
*/
--
1.7.5.4
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [RFC PATCH 12/13] Close files using notmuch_mailstore_close instead of fclose
2012-02-15 22:01 [RFC PATCH 00/13] Modular message store code Ethan Glasser-Camp
` (10 preceding siblings ...)
2012-02-15 22:02 ` [RFC PATCH 11/13] Add a close function to mailstore Ethan Glasser-Camp
@ 2012-02-15 22:02 ` Ethan Glasser-Camp
2012-02-15 22:02 ` [RFC PATCH 13/13] First crack at a CouchDB mailstore Ethan Glasser-Camp
` (2 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Ethan Glasser-Camp @ 2012-02-15 22:02 UTC (permalink / raw)
To: notmuch; +Cc: Ethan Glasser-Camp
From: Ethan Glasser-Camp <ethan@betacantrips.com>
This requires a little bit of juggling in lib/sha1.c. Wrapper
functions provide the FILE*. Instead of closing the file immediately
ourselves, we let the wrapper functions close it.
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
---
lib/message-file.c | 6 ++++--
lib/sha1.c | 14 ++++++++------
mime-node.c | 4 +++-
notmuch-show.c | 8 ++++----
4 files changed, 19 insertions(+), 13 deletions(-)
diff --git a/lib/message-file.c b/lib/message-file.c
index 61f4d04..51c77f9 100644
--- a/lib/message-file.c
+++ b/lib/message-file.c
@@ -34,6 +34,7 @@ typedef struct {
struct _notmuch_message_file {
/* File object */
+ notmuch_mailstore_t *mailstore;
FILE *file;
/* Header storage */
@@ -86,7 +87,7 @@ _notmuch_message_file_destructor (notmuch_message_file_t *message)
g_hash_table_destroy (message->headers);
if (message->file)
- fclose (message->file);
+ notmuch_mailstore_close (message->mailstore, message->file);
return 0;
}
@@ -105,6 +106,7 @@ _notmuch_message_file_open_ctx (void *ctx, notmuch_mailstore_t *mailstore,
talloc_set_destructor (message, _notmuch_message_file_destructor);
+ message->mailstore = mailstore;
message->file = notmuch_mailstore_open (mailstore, filename);
if (message->file == NULL)
goto FAIL;
@@ -363,7 +365,7 @@ notmuch_message_file_get_header (notmuch_message_file_t *message,
}
if (message->parsing_finished) {
- fclose (message->file);
+ notmuch_mailstore_close (message->mailstore, message->file);
message->file = NULL;
}
diff --git a/lib/sha1.c b/lib/sha1.c
index ea25999..4266720 100644
--- a/lib/sha1.c
+++ b/lib/sha1.c
@@ -87,7 +87,6 @@ _notmuch_sha1_of_filep (FILE *file)
if (feof (file)) {
break;
} else if (ferror (file)) {
- fclose (file);
return NULL;
}
} else {
@@ -99,8 +98,6 @@ _notmuch_sha1_of_filep (FILE *file)
result = _hex_of_sha1_digest (digest);
- fclose (file);
-
return result;
}
@@ -117,9 +114,12 @@ char *
notmuch_sha1_of_file (const char *filename)
{
FILE *file;
+ char *ret;
file = fopen (filename, "r");
- return _notmuch_sha1_of_filep (file);
+ ret = _notmuch_sha1_of_filep (file);
+ fclose (file);
+ return ret;
}
/* Create a hexadecimal string version of the SHA-1 digest of the
@@ -135,7 +135,9 @@ char *
notmuch_sha1_of_message (notmuch_mailstore_t *mailstore, const char *filename)
{
FILE *file;
-
+ char *ret;
file = notmuch_mailstore_open (mailstore, filename);
- return _notmuch_sha1_of_filep (file);
+ ret = _notmuch_sha1_of_filep (file);
+ notmuch_mailstore_close (mailstore, file);
+ return ret;
}
diff --git a/mime-node.c b/mime-node.c
index 856fc3b..db37189 100644
--- a/mime-node.c
+++ b/mime-node.c
@@ -27,6 +27,7 @@
typedef struct mime_node_context {
/* Per-message resources. These are allocated internally and must
* be destroyed. */
+ notmuch_mailstore_t *mailstore;
FILE *file;
GMimeStream *stream;
GMimeParser *parser;
@@ -54,7 +55,7 @@ _mime_node_context_free (mime_node_context_t *res)
g_object_unref (res->stream);
if (res->file)
- fclose (res->file);
+ notmuch_mailstore_close (res->mailstore, res->file);
return 0;
}
@@ -90,6 +91,7 @@ mime_node_open (const void *ctx, notmuch_message_t *message,
}
talloc_set_destructor (mctx, _mime_node_context_free);
+ mctx->mailstore = mailstore;
mctx->file = notmuch_mailstore_open (mailstore, filename);
if (! mctx->file) {
fprintf (stderr, "Error opening %s: %s\n", filename, strerror (errno));
diff --git a/notmuch-show.c b/notmuch-show.c
index 0d2a246..2e8e4c9 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -309,7 +309,7 @@ format_message_mbox (const void *ctx,
printf ("\n");
- fclose (file);
+ notmuch_mailstore_close (mailstore, file);
}
static void
@@ -966,18 +966,18 @@ do_show_single (void *ctx,
size = fread (buf, 1, sizeof (buf), file);
if (ferror (file)) {
fprintf (stderr, "Error: Read failed from %s\n", filename);
- fclose (file);
+ notmuch_mailstore_close (mailstore, file);
return 1;
}
if (fwrite (buf, size, 1, stdout) != 1) {
fprintf (stderr, "Error: Write failed\n");
- fclose (file);
+ notmuch_mailstore_close (mailstore, file);
return 1;
}
}
- fclose (file);
+ notmuch_mailstore_close (mailstore, file);
} else {
--
1.7.5.4
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [RFC PATCH 13/13] First crack at a CouchDB mailstore
2012-02-15 22:01 [RFC PATCH 00/13] Modular message store code Ethan Glasser-Camp
` (11 preceding siblings ...)
2012-02-15 22:02 ` [RFC PATCH 12/13] Close files using notmuch_mailstore_close instead of fclose Ethan Glasser-Camp
@ 2012-02-15 22:02 ` Ethan Glasser-Camp
2012-02-16 0:56 ` [RFC PATCH 00/13] Modular message store code Mark Walters
2012-02-16 7:47 ` Stewart Smith
14 siblings, 0 replies; 17+ messages in thread
From: Ethan Glasser-Camp @ 2012-02-15 22:02 UTC (permalink / raw)
To: notmuch; +Cc: Ethan Glasser-Camp
From: Ethan Glasser-Camp <ethan@betacantrips.com>
This introduces new parameters to notmuch-config to store the CouchDB
URL and the "name" of the database.
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
---
Makefile.local | 3 +
lib/mailstore.c | 109 ++++++++++++++++++++++++++++++++
notmuch-client.h | 14 ++++
notmuch-config.c | 91 +++++++++++++++++++++++++-
notmuch-new.c | 184 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
5 files changed, 397 insertions(+), 4 deletions(-)
diff --git a/Makefile.local b/Makefile.local
index 1131dea..a105e58 100644
--- a/Makefile.local
+++ b/Makefile.local
@@ -27,6 +27,9 @@ endif
UPSTREAM_TAG=$(subst ~,_,$(VERSION))
DEB_TAG=debian/$(UPSTREAM_TAG)-1
+# FIXME: Where should this really go?
+LDFLAGS += $(shell pkg-config --libs couchdb-glib-1.0 libsoup-2.4)
+extra_cflags += $(shell pkg-config --cflags couchdb-glib-1.0 libsoup-2.4)
RELEASE_HOST=notmuchmail.org
RELEASE_DIR=/srv/notmuchmail.org/www/releases
diff --git a/lib/mailstore.c b/lib/mailstore.c
index 51c2710..4d7cc79 100644
--- a/lib/mailstore.c
+++ b/lib/mailstore.c
@@ -18,6 +18,10 @@
#include <stdio.h>
#include <stdarg.h>
+#include <couchdb-session.h>
+#include <couchdb-database.h>
+#include <couchdb-document.h>
+#include <glib.h>
#include "notmuch-private.h"
@@ -58,6 +62,101 @@ _maildir_rename_function (unused (notmuch_mailstore_t *mailstore),
return rename (old_filename, new_filename);
}
+struct _couchdb_data {
+ char *db_path;
+ CouchdbDatabase *database;
+ GHashTable *files_to_documents;
+};
+
+/* CouchDB mailstore */
+static notmuch_status_t
+_couchdb_constructor (void **data, va_list ap)
+{
+ CouchdbSession *session = NULL;
+ CouchdbDatabase *database = NULL;
+ GError *error = NULL;
+ char *uri = NULL;
+ char *db_name = NULL;
+ struct _couchdb_data *my_data = NULL;
+
+ uri = va_arg (ap, char*);
+ session = couchdb_session_new (uri);
+
+ db_name = va_arg (ap, char*);
+ database = couchdb_session_get_database (session, db_name, &error);
+ if (database == NULL) {
+ fprintf (stderr, "Couldn't access database %s: %s\n", db_name,
+ error->message);
+ return NOTMUCH_STATUS_FILE_ERROR;
+ }
+
+ my_data = talloc_size (NULL, sizeof (struct _couchdb_data));
+ my_data->database = database;
+ my_data->db_path = va_arg (ap, char*);
+ my_data->files_to_documents = g_hash_table_new (NULL, NULL);
+ (*data) = (void*)my_data;
+
+ return NOTMUCH_STATUS_SUCCESS;
+}
+
+static FILE *
+_couchdb_open_function (notmuch_mailstore_t *mailstore,
+ const char *filename)
+{
+ CouchdbDatabase *database = NULL;
+ CouchdbDocument *document = NULL;
+ GError *error = NULL;
+ const char *text = NULL;
+ const char *relative = NULL;
+ struct _couchdb_data *data = (struct _couchdb_data *)mailstore->data;
+ FILE *ret = NULL;
+ database = data->database;
+ /* message assumes all files should be contained within db_path.
+ * This isn't true for us, so remove the db_path.
+ * I'd like to use _notmuch_database_relative_path but I don't have
+ * a notmuch_database_t*.
+ */
+ relative = filename;
+ if (strncmp (filename, data->db_path, strlen (data->db_path)) == 0) {
+ relative = filename + strlen (data->db_path);
+ while (*relative == '/' && *(relative+1) == '/')
+ relative++;
+ }
+
+ document = couchdb_database_get_document (database, relative, &error);
+ if (document == NULL)
+ /* file doesn't exist. Maybe it got deleted? */
+ return NULL;
+
+ text = couchdb_document_get_string_field (document, "text");
+ /* FIXME: null bytes in the mail file? */
+ ret = fmemopen ((char *)text, strlen(text), "r");
+ g_hash_table_insert (data->files_to_documents, ret, document);
+ return ret;
+}
+
+static int
+_couchdb_close_function (notmuch_mailstore_t *mailstore, FILE *file)
+{
+ struct _couchdb_data *data = (struct _couchdb_data *)mailstore->data;
+ GHashTable *hash = data->files_to_documents;
+ CouchdbDocument *document;
+ document = g_hash_table_lookup (hash, file);
+ g_object_unref (document);
+ fclose (file); /* just to be polite ;) */
+ g_hash_table_remove (hash, file);
+ return 0;
+}
+
+static int
+_couchdb_rename_function (unused (notmuch_mailstore_t *mailstore),
+ unused (const char *old_filename),
+ unused (const char *new_filename))
+{
+ /* Pass for now. */
+ return 0;
+}
+
/* A mailstore is defined as:
*
* - A function used to "open" a mail message. This takes the
@@ -80,12 +179,22 @@ notmuch_mailstore_maildir = { _maildir_constructor,
_maildir_rename_function,
NULL };
+_notmuch_mailstore
+notmuch_mailstore_couchdb = { _couchdb_constructor,
+ _couchdb_open_function, _couchdb_close_function,
+ _couchdb_rename_function,
+ NULL};
+
+
_notmuch_mailstore *
notmuch_mailstore_get_by_name (const char *name)
{
if (strcmp (name, "maildir") == 0)
return ¬much_mailstore_maildir;
+ if (strcmp (name, "couchdb") == 0)
+ return ¬much_mailstore_couchdb;
+
return NULL;
}
diff --git a/notmuch-client.h b/notmuch-client.h
index 405aad7..12dc868 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -230,6 +230,20 @@ void
notmuch_config_set_database_type (notmuch_config_t *config,
const char *database_type);
+const char *
+notmuch_config_get_database_uri (notmuch_config_t *config);
+
+void
+notmuch_config_set_database_uri (notmuch_config_t *config,
+ const char *database_uri);
+
+const char *
+notmuch_config_get_database_name (notmuch_config_t *config);
+
+void
+notmuch_config_set_database_name (notmuch_config_t *config,
+ const char *database_name);
+
notmuch_mailstore_t *
notmuch_config_get_mailstore (notmuch_config_t *config);
diff --git a/notmuch-config.c b/notmuch-config.c
index 99f872d..6090150 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -37,8 +37,8 @@ static const char database_config_comment[] =
"\n"
" The following options are supported here:\n"
"\n"
- "\ttype The type of mail backend. The only currently supported\n"
- "\t value is \"maildir\".\n"
+ "\ttype The type of mail backend. The currently supported\n"
+ "\t values are \"maildir\" and \"couchdb\".\n"
"\tpath For the maildir backend, the top-level maildir directory.\n"
"\t For all backends, the location where notmuch should store its\n"
"\t database. Notmuch will store its database within a sub-directory\n"
@@ -49,7 +49,14 @@ static const char database_config_comment[] =
" This backend reads mail from a directory tree where files are\n"
" individual email messages.\n"
" The only configuration option is 'path' which should be the top-level\n"
- " directory.\n";
+ " directory.\n"
+ " CouchDB backend\n"
+ "\n"
+ " This backend reads mail from a CouchDB database via HTTP.\n"
+ " For more details on the setup of such a database, please see the help\n"
+ " files.\n"
+ " The configuration options are 'uri' and 'name', which specify the URI\n"
+ " of the CouchDB instance and the database name of the mail store.\n";
static const char new_config_comment[] =
" Configuration for \"notmuch new\"\n"
@@ -113,6 +120,8 @@ struct _notmuch_config {
char *database_path;
char *database_type;
+ char *database_uri;
+ char *database_name;
char *user_name;
char *user_primary_email;
const char **user_other_email;
@@ -273,6 +282,8 @@ notmuch_config_open (void *ctx,
config->database_path = NULL;
config->database_type = NULL;
+ config->database_uri = NULL;
+ config->database_name = NULL;
config->user_name = NULL;
config->user_primary_email = NULL;
config->user_other_email = NULL;
@@ -339,6 +350,12 @@ notmuch_config_open (void *ctx,
notmuch_config_set_database_type (config, "maildir");
}
+ if (notmuch_config_get_database_uri (config) == NULL)
+ notmuch_config_set_database_uri (config, "");
+
+ if (notmuch_config_get_database_name (config) == NULL)
+ notmuch_config_set_database_name (config, "");
+
if (notmuch_config_get_user_name (config) == NULL) {
char *name = get_name_from_passwd_file (config);
notmuch_config_set_user_name (config, name);
@@ -584,6 +601,62 @@ notmuch_config_set_database_type (notmuch_config_t *config,
config->database_type = NULL;
}
+const char *
+notmuch_config_get_database_uri (notmuch_config_t *config)
+{
+ char *uri;
+
+ if (config->database_uri == NULL) {
+ uri = g_key_file_get_string (config->key_file,
+ "database", "uri", NULL);
+ if (uri) {
+ config->database_uri = talloc_strdup (config, uri);
+ free (uri);
+ }
+ }
+
+ return config->database_uri;
+}
+
+void
+notmuch_config_set_database_uri (notmuch_config_t *config,
+ const char *database_uri)
+{
+ g_key_file_set_string (config->key_file,
+ "database", "uri", database_uri);
+
+ talloc_free (config->database_uri);
+ config->database_uri = NULL;
+}
+
+const char *
+notmuch_config_get_database_name (notmuch_config_t *config)
+{
+ char *name;
+
+ if (config->database_name == NULL) {
+ name = g_key_file_get_string (config->key_file,
+ "database", "name", NULL);
+ if (name) {
+ config->database_name = talloc_strdup (config, name);
+ free (name);
+ }
+ }
+
+ return config->database_name;
+}
+
+void
+notmuch_config_set_database_name (notmuch_config_t *config,
+ const char *database_name)
+{
+ g_key_file_set_string (config->key_file,
+ "database", "name", database_name);
+
+ talloc_free (config->database_name);
+ config->database_name = NULL;
+}
+
notmuch_mailstore_t *
notmuch_config_get_mailstore (notmuch_config_t *config)
{
@@ -595,7 +668,17 @@ notmuch_config_get_mailstore (notmuch_config_t *config)
notmuch_status_t status;
const char *type = notmuch_config_get_database_type (config);
notmuch_mailstore_t *mailstore = notmuch_mailstore_get_by_name (type);
- status = notmuch_mailstore_construct (mailstore);
+ if (strcmp (type, "maildir") == 0)
+ status = notmuch_mailstore_construct (mailstore);
+ else if (strcmp (type, "couchdb") == 0)
+ status = notmuch_mailstore_construct (mailstore,
+ notmuch_config_get_database_uri (config),
+ notmuch_config_get_database_name (config),
+ notmuch_config_get_database_path (config));
+ else
+ /* Doomed, doomed, doomed */
+ status = NOTMUCH_STATUS_FILE_ERROR;
+
if (status != NOTMUCH_STATUS_SUCCESS) {
/* abort messily? */
}
diff --git a/notmuch-new.c b/notmuch-new.c
index d30fba1..3c1acb2 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -21,6 +21,11 @@
#include "notmuch-client.h"
#include <unistd.h>
+#include <libsoup/soup-method.h>
+#include <couchdb-session.h>
+#include <couchdb-database.h>
+#include <couchdb-document.h>
+#include <json-glib/json-glib.h>
typedef struct _filename_node {
char *filename;
@@ -297,6 +302,182 @@ _add_message (add_files_state_t *state, notmuch_database_t *notmuch,
return ret;
}
+/* Send an unsupported message to a couchdb instance.
+ *
+ * This function is "supposed" to be "part" of the "public API",
+ * but it isn't declared in couchdb-glib's header files. See:
+ * https://bugs.launchpad.net/couchdb-glib/+bug/927847
+ */
+gboolean
+couchdb_session_send_message (CouchdbSession *session, const char *method, const char *url, const char *body, JsonParser *output, GError **error);
+
+/* Process a JSON "change" object and either add or delete the "file".
+ *
+ * This is based on code from couchdb-glib, which is why it's a weird
+ * melange of glib style and notmuch style.
+ *
+ * As with Maildir, we assume that message objects never change.
+ */
+static void
+couchdb_process_change (add_files_state_t *state,
+ notmuch_database_t *notmuch,
+ CouchdbDatabase *database,
+ JsonNode *node)
+{
+ JsonObject *this_change;
+ const gchar *id;
+ CouchdbDocument *document;
+ GError *error = NULL;
+
+ if (json_node_get_node_type (node) != JSON_NODE_OBJECT)
+ return;
+
+ this_change = json_node_get_object (node);
+ if (!json_object_has_member (this_change, "id"))
+ return;
+
+ id = json_object_get_string_member (this_change, "id");
+
+ /* We need to try retrieving the document, to check if it's removed or not */
+ document = couchdb_database_get_document (database, id, &error);
+ if (document) {
+ /* We got a document, dump it into Notmuch */
+ _report_before_adding_file (state, id);
+ _add_message (state, notmuch, id);
+ _report_added_file (state);
+ g_object_unref (G_OBJECT (document));
+ }
+ else {
+ if (error != NULL) {
+ g_warning ("Error retrieving document '%s': %s", id, error->message);
+ g_error_free (error);
+ } else {
+ /* The document is no longer in the DB, notify */
+ id = talloc_strdup (state->removed_files, id);
+ _filename_list_add (state->removed_files, id);
+ }
+ }
+}
+
+/* Fetch a batch of database updates from couch's "changes" feed.
+ *
+ * This is essentially a copied and modified version of code from
+ * couchdb-glib. There's code to "watch a feed of changes", but I just
+ * want to do it once and synchronously.
+ */
+static notmuch_status_t
+couchdb_add_messages_batch (add_files_state_t *state,
+ notmuch_database_t *notmuch,
+ CouchdbDatabase *database,
+ guint32 *last_seq, int limit)
+{
+ char *url;
+ JsonParser *parser;
+ GError *error = NULL;
+
+ url = g_strdup_printf ("%s/%s/_changes?since=%d&limit=%d",
+ couchdb_session_get_uri (couchdb_database_get_session (database)),
+ couchdb_database_get_name (database),
+ *last_seq, limit);
+ parser = json_parser_new ();
+
+ if (couchdb_session_send_message (couchdb_database_get_session (database),
+ SOUP_METHOD_GET, url, NULL, parser,
+ &error)) {
+ JsonNode *root_node;
+
+ root_node = json_parser_get_root (parser);
+ if (json_node_get_node_type (root_node) == JSON_NODE_OBJECT) {
+ JsonObject *root_object;
+ JsonArray *results;
+
+ root_object = json_node_get_object (root_node);
+ results = json_object_get_array_member (root_object, "results");
+ if (results) {
+ GList *json_elements, *sl;
+
+ json_elements = json_array_get_elements (results);
+ for (sl = json_elements; !interrupted && sl != NULL; sl = sl->next)
+ couchdb_process_change (state, notmuch, database,
+ (JsonNode *) sl->data);
+ g_list_free (json_elements);
+ }
+
+ if (json_object_has_member (root_object, "last_seq"))
+ *last_seq = json_object_get_int_member (root_object, "last_seq");
+ }
+ }
+
+ /* Free memory */
+ g_object_unref (G_OBJECT (parser));
+ g_free (url);
+
+ return NOTMUCH_STATUS_SUCCESS;
+}
+
+/* Couchdb add_files function.
+ *
+ * Use the Couchdb _changes API to just ask what files have been added or deleted.
+ *
+ * We use a dummy "/" directory to store the last change we got from couch.
+ */
+static notmuch_status_t
+couchdb_add_files (notmuch_database_t *notmuch,
+ notmuch_config_t *config,
+ add_files_state_t *state)
+{
+ CouchdbSession *session;
+ CouchdbDatabase *database;
+ GError *error;
+ notmuch_directory_t *directory;
+ notmuch_status_t status;
+ time_t db_mtime;
+ guint32 last_seq = 0;
+ guint32 old_last_seq = 0;
+ const char *db_name;
+ const char *uri;
+
+ /* These are probably abstraction-breaking hacks. Life is tough. */
+ uri = notmuch_config_get_database_uri (config);
+ db_name = notmuch_config_get_database_name (config);
+
+ /* FIXME: is this necessary? I think probably not? */
+ /*
+ db_name = talloc_strdup (config, db_name);
+ uri = talloc_strdup (config, uri);
+ */
+
+ session = couchdb_session_new (uri);
+ database = couchdb_session_get_database (session, db_name, &error);
+ if (database == NULL) {
+ fprintf (stderr, "Error: couldn't access couchdb database %s, %s: %s",
+ uri, db_name, error->message);
+ return NOTMUCH_STATUS_FILE_ERROR;
+ }
+
+ /* Store a dummy directory at / that just contains last_seq as its mtime. */
+ directory = notmuch_database_get_directory (notmuch, "/");
+ db_mtime = notmuch_directory_get_mtime (directory);
+ last_seq = (int)db_mtime;
+
+ /* Grab updates in sets of 100 just to be safe with memory. */
+ do {
+ if (interrupted)
+ break;
+ old_last_seq = last_seq;
+ status = couchdb_add_messages_batch (state, notmuch, database,
+ &last_seq, 100);
+ if (status != NOTMUCH_STATUS_SUCCESS) {
+ return status;
+ }
+ } while (last_seq == old_last_seq + 100);
+
+ notmuch_directory_set_mtime (directory, last_seq);
+ g_object_unref (database);
+ g_object_unref (session);
+ notmuch_directory_destroy (directory);
+ return NOTMUCH_STATUS_SUCCESS;
+}
/* Examine 'path' recursively as follows:
*
@@ -678,6 +859,9 @@ add_files (notmuch_database_t *notmuch, notmuch_config_t *config,
if (strcmp (notmuch_config_get_database_type (config), "maildir") == 0)
return maildir_add_files (notmuch, path, state);
+ else if (strcmp (notmuch_config_get_database_type (config), "couchdb") == 0)
+ return couchdb_add_files (notmuch, config, state);
+
/* Default case */
fprintf (stderr, "Could not add files for mailstore %s: unknown mailstore\n",
notmuch_config_get_database_type (config));
--
1.7.5.4
^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [RFC PATCH 00/13] Modular message store code
2012-02-15 22:01 [RFC PATCH 00/13] Modular message store code Ethan Glasser-Camp
` (12 preceding siblings ...)
2012-02-15 22:02 ` [RFC PATCH 13/13] First crack at a CouchDB mailstore Ethan Glasser-Camp
@ 2012-02-16 0:56 ` Mark Walters
2012-03-01 13:51 ` Ethan Glasser-Camp
2012-02-16 7:47 ` Stewart Smith
14 siblings, 1 reply; 17+ messages in thread
From: Mark Walters @ 2012-02-16 0:56 UTC (permalink / raw)
To: Ethan Glasser-Camp, notmuch
On Wed, 15 Feb 2012 17:01:53 -0500, Ethan Glasser-Camp <glasse@cs.rpi.edu> wrote:
> Hi guys,
>
> I'm submitting as RFC this patch series, which introduces the idea of
> a "mailstore", a "class" that defines how to access mail, instead of
> currently assuming it's always some Maildir-ish hierarchy that
> contains a bunch of mail.
>
> This was listed as a wishlist item on
> http://notmuchmail.org/feature-requests/, so I went ahead and took a
> crack at it, learning a lot about the codebase as I did. I'm sure
> there are tons of stylistic concerns so I'd like to get as much
> feedback as possible, starting of course with "Does a feature like
> this have a chance of ever making it in" and followed by "Am I on the
> right track".
>
> Note that this series breaks the language bundings; the Python
> bindings have very minimal tests so I very minimally fixed them
> (probably still broken in other ways), but the Ruby and Go bindings
> are probably super broken. Note also that the one message = one file
> approach is pretty thoroughly embedded into Notmuch and there are lots
> of places (again such as the Python bindings) where this has yet to be
> rooted out.
>
> They say an interface isn't trustworthy until you've implemented it
> three times, so while most of the patches define the interface, the
> last patch adds support for an experimental CouchDB backend. It's got
> at least one known bug (it indexes everything, whether or not it's a
> mail object), sometimes it hangs when trying to access a message, and
> it's definitely leaking memory in notmuch-new. I haven't done strict
> timing comparisons but it seems like notmuch-search and notmuch-show
> are approximately as fast as with maildir and notmuch-new takes maybe
> 25% longer. Nevertheless, it is included as a demonstration that the
> interface is at least plausible.
>
> The implementation of "mailstores" follows these principles:
>
> - Messages still have "filenames", but the mailstore gets to define
> its own semantics for these filenames (document ids, file + byte
> offset..). _notmuch_message_ensure_filename_list converts all
> filenames coming out of the DB to be absolute paths centered at the
> user's database path, which is inconvenient for Couchdb, but workable.
Obviously I have not looked at the patch set in detail yet but I have a
quick question. Since you are allowing more general filenames anyway
couldn't you encode mailstore in filename? Eg
mbox://some-path[:byte-postion], or "imap://server..."
This would allow lots of different types of mailstore to be used
concurrently, and would push all the mailstore knowledge down into the
file handling functions and away from the callers of file handling
functions.
Of course there may be lots of good reasons why this doesn't work.
Best wishes
Mark
> - The user keeps all mail in one mailstore. The alternative, which is
> that each message can be in a different mailstore, seemed like a lot
> more work. "One mailstore" makes sense when it's cases like maildir
> vs. couchdb, but if we decide to introduce a "mbox" backend --
> directory tree with mbox files -- then it might "overlap" with the
> maildir mailstore, and then who knows?
>
> Patch 1 introduces the configuration parameter database.type, which
> will be used to select a mailstore type.
>
> Patch 2 introduces the most important API for a mailstore, notmuch_mailstore_open, and makes it required when creating a message_file. Patch 3 fixes the Python breakage this creates.
>
> Patch 4-6 replace the other places where files are opened directly with calls to notmuch_mailstore_open.
>
> Patch 7-8 prepare notmuch-new to be more generic. I couldn't find an elegant way to combine add_files logic with other mailstores, so I just decided each mailstore should have its own add_files function.
>
> Patches 9-11 add other functions to the mailstore API -- to rename files, to close files, and to "construct" a mailstore. Patch 12 uses the "close" API to close files (where necessary).
>
> Patch 13 proposes the CouchDB mailstore as one block of code.
>
> Points to address:
>
> - Where to put the new notmuch_mailstore_t* parameter in all these functions? I applied a "decreasing order of importance" heuristic, but it's a little weird in places like notmuch_database_open and notmuch_database_create.
>
> - Is there a better, more elegant way to pass around mailstore objects without adding parameters to each function? Additionally, should I be using some other class-like mechanism for mailstores instead of hacks involving structs?
>
> - Should this be configured under [database] or perhaps under some other new heading?
>
> - How strict is the rule that braces shouldn't be there if the body of a loop/conditional is only one line? This feels really strange to me coming from Python.
>
> - If I'm already touching code, should I add other drive-by fixes, as in patch 05, or should I resolutely refuse to change anything, as in patch 07?
>
> - Should something like the CouchDB backend be optional, and if so, what mechanisms do I need to use to make that happen?
>
> Thanks so much for your time!
> Ethan
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [RFC PATCH 00/13] Modular message store code
2012-02-15 22:01 [RFC PATCH 00/13] Modular message store code Ethan Glasser-Camp
` (13 preceding siblings ...)
2012-02-16 0:56 ` [RFC PATCH 00/13] Modular message store code Mark Walters
@ 2012-02-16 7:47 ` Stewart Smith
14 siblings, 0 replies; 17+ messages in thread
From: Stewart Smith @ 2012-02-16 7:47 UTC (permalink / raw)
To: Ethan Glasser-Camp, notmuch
On Wed, 15 Feb 2012 17:01:53 -0500, Ethan Glasser-Camp <glasse@cs.rpi.edu> wrote:
> I'm submitting as RFC this patch series, which introduces the idea of
> a "mailstore", a "class" that defines how to access mail, instead of
> currently assuming it's always some Maildir-ish hierarchy that
> contains a bunch of mail.
This is really awesome.
Quite a while ago now I did some experiments on storing my entire
Maildir inside git packs instead of in maildir. This produced an
*amazing* saving in disk space used. My idea is to end up with Maildir
for "current" (as everything delivers into Maildir without a problem)
and then on a (say) monthly basis, packing all mail into an archive file
and have notmuch be able to still read it.
you know what... this patch set has re-ignited my interest in making
that work.
--
Stewart Smith
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [RFC PATCH 00/13] Modular message store code
2012-02-16 0:56 ` [RFC PATCH 00/13] Modular message store code Mark Walters
@ 2012-03-01 13:51 ` Ethan Glasser-Camp
0 siblings, 0 replies; 17+ messages in thread
From: Ethan Glasser-Camp @ 2012-03-01 13:51 UTC (permalink / raw)
To: Mark Walters; +Cc: notmuch
On 02/15/2012 07:56 PM, Mark Walters wrote:
> Obviously I have not looked at the patch set in detail yet but I have a
> quick question. Since you are allowing more general filenames anyway
> couldn't you encode mailstore in filename? Eg
> mbox://some-path[:byte-postion], or "imap://server..."
>
> This would allow lots of different types of mailstore to be used
> concurrently, and would push all the mailstore knowledge down into the
> file handling functions and away from the callers of file handling
> functions.
>
> Of course there may be lots of good reasons why this doesn't work.
>
Hi, sorry for the delay.
As far as I can tell, currently notmuch stores message filenames in
Xapian as paths relative to the top-level maildir. I think this is done
so that the maildir can be moved and, if the .notmuch-config is updated,
mails are correctly detected and not duplicated. This would be
especially important when you're talking about changing IMAP servers or
CouchDB instances.
If I wanted to preserve this feature, the URIs stored as filenames would
have to be relative to a given mailstore. For example,
maildir://maildir-1/INBOX/some-filename could mean the file
INBOX/some-filename in a maildir at /home/user/some-maildir. But then
this raises the two following issues:
- How does information about mailstores -- for example, that maildir-1
=> /home/user/some-maildir -- enter the library? Do we stick all of that
information in notmuch_database_t, and then pass a reference to it in
notmuch_message_file_open? Perhaps a global
notmuch_mailstore_register(name, parameters..) registry? Or maybe a
notmuch_mailstore_info type that gets passed around similarly to the
mailstore type in this patch set?
- Do we mandate that all the filenames in the database be updated or do
we just assume non-URI-style filenames are relative to some "default"
mailstore?
All of which is a fancy way of saying I haven't had the time to write
the code necessary to explore this idea but think something like it will
be necessary to support the obviously-valuable feature of multiple
mailstores. Depending on your answer to the first question, I guess the
patch series might or might not be a useful starting point.
Thanks for your feedback,
Ethan
^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2012-03-01 13:51 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-02-15 22:01 [RFC PATCH 00/13] Modular message store code Ethan Glasser-Camp
2012-02-15 22:01 ` [RFC PATCH 01/13] Create configuration paramater database.type Ethan Glasser-Camp
2012-02-15 22:01 ` [RFC PATCH 02/13] Add the concept of a mailstore in its absolute minimal sense Ethan Glasser-Camp
2012-02-15 22:01 ` [RFC PATCH 03/13] Introduce mailstore in the python bindings Ethan Glasser-Camp
2012-02-15 22:01 ` [RFC PATCH 04/13] Replace remaining places where fopen occurs Ethan Glasser-Camp
2012-02-15 22:01 ` [RFC PATCH 05/13] notmuch_database_add_message calculates sha1 of messages using mailstore Ethan Glasser-Camp
2012-02-15 22:01 ` [RFC PATCH 06/13] Pass mailstore to _notmuch_message_index_file Ethan Glasser-Camp
2012-02-15 22:02 ` [RFC PATCH 07/13] notmuch-new: pull out useful bits of add_files_recursive Ethan Glasser-Camp
2012-02-15 22:02 ` [RFC PATCH 08/13] count_files and add_files shall be generic Ethan Glasser-Camp
2012-02-15 22:02 ` [RFC PATCH 09/13] Mailstore also provides a "rename" function Ethan Glasser-Camp
2012-02-15 22:02 ` [RFC PATCH 10/13] Introduce concept of mailstore "constructor" Ethan Glasser-Camp
2012-02-15 22:02 ` [RFC PATCH 11/13] Add a close function to mailstore Ethan Glasser-Camp
2012-02-15 22:02 ` [RFC PATCH 12/13] Close files using notmuch_mailstore_close instead of fclose Ethan Glasser-Camp
2012-02-15 22:02 ` [RFC PATCH 13/13] First crack at a CouchDB mailstore Ethan Glasser-Camp
2012-02-16 0:56 ` [RFC PATCH 00/13] Modular message store code Mark Walters
2012-03-01 13:51 ` Ethan Glasser-Camp
2012-02-16 7:47 ` Stewart Smith
Code repositories for project(s) associated with this public inbox
https://yhetil.org/notmuch.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).