* Mailstore abstraction & maildir synchronization
@ 2010-03-18 15:39 Michal Sojka
2010-03-18 15:39 ` [PATCH 1/4] Mailstore abstraction interface Michal Sojka
` (4 more replies)
0 siblings, 5 replies; 25+ messages in thread
From: Michal Sojka @ 2010-03-18 15:39 UTC (permalink / raw)
To: notmuch
Hi all,
I've finally found some time to implement the mailstore abstraction
was initially described in id:87ljecmnbd.fsf@steelpick.localdomain and
id:87eijqlz54.fsf@steelpick.localdomain.
The idea is to allow notmuch operate on different types of mail
storage (e.g. mail in git repositories) and to store the tags in the
storage together with mails. The aim is the ability to synchronize
mails and tags between computers. The following patch series is the
first version which I'm able to use on daily basis so I'd like to get
some feedback.
In the current form, the patch series implements two mail stores:
1. plain files (compatible with the current notmuch)
2. maildir (synchronizes certain tags with maildir flags)
The series passes the test suite. For the maildir store, there are
additional tests, but you need need the modularized testsuite (taken
from git). The whole patch series is also available from
git://rtime.felk.cvut.cz/notmuch.git mailstore-abstraction-v1 (this
branch wont be rebased).
Known bugs and limitations:
- Only file-based storage is suported. Notmuch access the files
directly, and not via the mailstore interface.
- (maildir) Viewing/storing of attachments of unread messages doesn't
work. The reason is that when you view the message it its unread tag
is removed which leads to rename of the file, but Emacs still uses
the original name to access the attachment.
Workaround: close the message and open it again.
Maildir howto:
1. Backup you emails
2. Apply the patches (at least 1-3)
3. Configure notmuch to use maildir store
cat > ~/.notmuch-config <<EOF
[mailstore]
type=maildir
EOF
4. Enjoy
Advantages for me:
- Whenever I read a message my mobile phone gets that information
since I run offlineimap periodically on background.
- My mailsync script (notmuch new + tagging) is much faster because it
does not call offlineimap and notmuchsync (two times). I call it
manually, because I want to decide when new mail should appear.
Cheers,
-Michal
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH 1/4] Mailstore abstraction interface
2010-03-18 15:39 Mailstore abstraction & maildir synchronization Michal Sojka
@ 2010-03-18 15:39 ` Michal Sojka
2010-03-18 15:39 ` [PATCH 2/4] Convert mailstore abstraction Michal Sojka
` (3 subsequent siblings)
4 siblings, 0 replies; 25+ messages in thread
From: Michal Sojka @ 2010-03-18 15:39 UTC (permalink / raw)
To: notmuch
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 23524 bytes --]
The goal of mailstore abstraction is to allow notmuch to store tags
together with email messages. The abstract interface is needed because
people want to use different ways of storing their emails. This
patchseries implements two types of mailstore - plain files and
maildir. It is expected that additional git-based mailstore will
follow.
This patch contains only the interface changes. No functionality is
added, removed or changed.
A new configuration group [mailstore] is defined. Currently, there is
only one key 'type' whose value determines the used mailstore. The
default value of this option is the plain-file mailstore currently
implemented in notmuch.
Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz>
---
lib/Makefile.local | 2 +
lib/database-private.h | 1 +
lib/database.cc | 15 ++++++--
lib/mailstore-files.c | 43 ++++++++++++++++++++++++
lib/mailstore-private.h | 59 ++++++++++++++++++++++++++++++++
lib/mailstore.c | 77 ++++++++++++++++++++++++++++++++++++++++++
lib/message.cc | 13 +++++++
lib/notmuch.h | 85 ++++++++++++++++++++++++++++++++++++++++++++--
notmuch-client.h | 7 ++++
notmuch-config.c | 34 +++++++++++++++++++
notmuch-count.c | 3 +-
notmuch-dump.c | 3 +-
notmuch-new.c | 6 ++-
notmuch-reply.c | 3 +-
notmuch-restore.c | 3 +-
notmuch-search-tags.c | 3 +-
notmuch-search.c | 3 +-
notmuch-show.c | 3 +-
notmuch-tag.c | 3 +-
19 files changed, 348 insertions(+), 18 deletions(-)
create mode 100644 lib/mailstore-files.c
create mode 100644 lib/mailstore-private.h
create mode 100644 lib/mailstore.c
diff --git a/lib/Makefile.local b/lib/Makefile.local
index 495b27e..fc8883d 100644
--- a/lib/Makefile.local
+++ b/lib/Makefile.local
@@ -3,6 +3,8 @@ extra_cflags += -I$(dir)
libnotmuch_c_srcs = \
$(dir)/libsha1.c \
+ $(dir)/mailstore.c \
+ $(dir)/mailstore-files.c\
$(dir)/message-file.c \
$(dir)/messages.c \
$(dir)/sha1.c \
diff --git a/lib/database-private.h b/lib/database-private.h
index 41918d7..4499b1a 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -49,6 +49,7 @@ struct _notmuch_database {
Xapian::TermGenerator *term_gen;
Xapian::ValueRangeProcessor *value_range_processor;
+ notmuch_mailstore_t *mailstore;
};
/* Convert tags from Xapian internal format to notmuch format.
diff --git a/lib/database.cc b/lib/database.cc
index c91e97c..93c8d0f 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -19,6 +19,7 @@
*/
#include "database-private.h"
+#include "mailstore-private.h"
#include <iostream>
@@ -438,7 +439,7 @@ parse_references (void *ctx,
}
notmuch_database_t *
-notmuch_database_create (const char *path)
+notmuch_database_create (const char *path, notmuch_mailstore_t *mailstore)
{
notmuch_database_t *notmuch = NULL;
char *notmuch_path = NULL;
@@ -474,7 +475,8 @@ notmuch_database_create (const char *path)
}
notmuch = notmuch_database_open (path,
- NOTMUCH_DATABASE_MODE_READ_WRITE);
+ NOTMUCH_DATABASE_MODE_READ_WRITE,
+ mailstore);
notmuch_database_upgrade (notmuch, NULL, NULL);
DONE:
@@ -497,7 +499,8 @@ _notmuch_database_ensure_writable (notmuch_database_t *notmuch)
notmuch_database_t *
notmuch_database_open (const char *path,
- notmuch_database_mode_t mode)
+ notmuch_database_mode_t mode,
+ notmuch_mailstore_t *mailstore)
{
notmuch_database_t *notmuch = NULL;
char *notmuch_path = NULL, *xapian_path = NULL;
@@ -605,6 +608,9 @@ notmuch_database_open (const char *path,
prefix_t *prefix = &PROBABILISTIC_PREFIX[i];
notmuch->query_parser->add_prefix (prefix->name, prefix->prefix);
}
+
+ notmuch->mailstore = mailstore;
+ mailstore->notmuch = notmuch;
} catch (const Xapian::Error &error) {
fprintf (stderr, "A Xapian exception occurred opening database: %s\n",
error.get_msg().c_str());
@@ -1493,7 +1499,8 @@ notmuch_database_add_message (notmuch_database_t *notmuch,
DONE:
if (message) {
- if (ret == NOTMUCH_STATUS_SUCCESS && message_ret)
+ if ((ret == NOTMUCH_STATUS_SUCCESS ||
+ ret == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) && message_ret)
*message_ret = message;
else
notmuch_message_destroy (message);
diff --git a/lib/mailstore-files.c b/lib/mailstore-files.c
new file mode 100644
index 0000000..92d7f5d
--- /dev/null
+++ b/lib/mailstore-files.c
@@ -0,0 +1,43 @@
+/* mailstore-files.c - Original notmuch mail store - a collection of
+ * plain-text email messages (one message per file).
+ *
+ * Copyright © 2009 Carl Worth
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see http://www.gnu.org/licenses/ .
+ *
+ * Authors: Carl Worth <cworth@cworth.org>
+ * Michal Sojka <sojkam1@fel.cvut.cz>
+ */
+
+#include "notmuch.h"
+#include "mailstore-private.h"
+
+
+static FILE *
+open_file(notmuch_mailstore_t *mailstore, const char *filename)
+{
+ const char *db_path, *abs_filename;
+
+ db_path = notmuch_database_get_path(mailstore->notmuch);
+ abs_filename = talloc_asprintf(mailstore, "%s/%s", db_path, filename);
+ if (unlikely(abs_filename == NULL))
+ return NULL;
+ return fopen (abs_filename, "r");
+}
+
+/* Original notmuch mail store */
+struct _notmuch_mailstore mailstore_files = {
+ .type = "files",
+ .open_file = open_file,
+};
diff --git a/lib/mailstore-private.h b/lib/mailstore-private.h
new file mode 100644
index 0000000..f606e3f
--- /dev/null
+++ b/lib/mailstore-private.h
@@ -0,0 +1,59 @@
+/* mailstore-private.h - Mailstore abstraction
+ *
+ * Copyright © 2010 Michal Sojka
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see http://www.gnu.org/licenses/ .
+ *
+ * Author: Michal Sojka <sojkam1@fel.cvut.cz>
+ */
+
+#ifndef MAILSTORE_PRIVATE_H
+#define MAILSTORE_PRIVATE_H
+
+#include "notmuch-private.h"
+
+struct _notmuch_mailstore {
+ /* Constant fields */
+ const char *type; /* identification in .notmuch-config */
+ void *priv;
+
+ /* Count files in mailstore so that initial invocation of notmuch
+ * new can report remaining time */
+ void (*count_files)(notmuch_mailstore_t *mailstore,
+ const char *path, int *count,
+ volatile sig_atomic_t *interrupted);
+ /* Scan for new messages and add them to the database. */
+ notmuch_private_status_t (*index_new)(notmuch_mailstore_t *mailstore,
+ const char* path,
+ notmuch_indexing_context_t *ctx);
+ /* Store the tags assigned to the 'messages' to the
+ * 'mailstore'. */
+ notmuch_private_status_t (*sync_tags)(notmuch_mailstore_t *mailstore,
+ notmuch_message_t *message);
+ /* Return file handle from which the message can be read.
+ *
+ * Currently, this is not used and all message access is hardcoded
+ * to notmuch. We will need this operation for git-based and other
+ * non-filesystem mailstores. */
+ FILE * (*open_file)(notmuch_mailstore_t *mailstore,
+ const char *filename);
+
+ /* Run-time fields */
+ notmuch_database_t *notmuch;
+};
+
+extern struct _notmuch_mailstore mailstore_files;
+extern struct _notmuch_mailstore mailstore_maildir;
+
+#endif /* MAILSTORE_PRIVATE_H */
diff --git a/lib/mailstore.c b/lib/mailstore.c
new file mode 100644
index 0000000..cdac9a6
--- /dev/null
+++ b/lib/mailstore.c
@@ -0,0 +1,77 @@
+/* mailstore.c - Mailstore abstraction
+ *
+ * Copyright © 2010 Michal Sojka
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see http://www.gnu.org/licenses/ .
+ *
+ * Author: Michal Sojka <sojkam1@fel.cvut.cz>
+ */
+
+#include <notmuch.h>
+#include "mailstore-private.h"
+
+#define ARRAY_SIZE(arr) (sizeof (arr) / sizeof (arr[0]))
+
+static notmuch_mailstore_t *available[] = {
+ &mailstore_files,
+};
+
+notmuch_mailstore_t *
+notmuch_mailstore_get_by_type (const char *type)
+{
+ unsigned i;
+
+ if (type == NULL)
+ return available[0];
+
+ for (i = 0; i < ARRAY_SIZE(available); i++) {
+ if (strcasecmp (type, available[i]->type) == 0)
+ return available[i];
+ }
+ return NULL;
+}
+
+const char *
+notmuch_mailstore_get_type (notmuch_mailstore_t *mailstore)
+{
+ if (!mailstore)
+ return NULL;
+
+ return mailstore->type;
+}
+
+void
+notmuch_mailstore_count_files (notmuch_mailstore_t *mailstore,
+ const char *path, int *count,
+ volatile sig_atomic_t *interrupted)
+{
+ mailstore->count_files (mailstore, path, count, interrupted);
+}
+
+notmuch_status_t
+notmuch_mailstore_index_new (notmuch_mailstore_t *mailstore,
+ const char *path,
+ notmuch_indexing_context_t *ctx)
+{
+ notmuch_private_status_t status;
+ status = mailstore->index_new (mailstore, path, ctx);
+ return COERCE_STATUS (status, "Index new");
+}
+
+FILE *
+notmuch_mailstore_open_file (notmuch_mailstore_t *mailstore,
+ const char *filename)
+{
+ return mailstore->open_file (mailstore, filename);
+}
diff --git a/lib/message.cc b/lib/message.cc
index 721c9a6..c32ee7d 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -20,6 +20,7 @@
#include "notmuch-private.h"
#include "database-private.h"
+#include "mailstore-private.h"
#include <stdint.h>
@@ -555,10 +556,22 @@ void
_notmuch_message_sync (notmuch_message_t *message)
{
Xapian::WritableDatabase *db;
+ notmuch_private_status_t status;
if (message->notmuch->mode == NOTMUCH_DATABASE_MODE_READ_ONLY)
return;
+ if (message->notmuch->mailstore->sync_tags) {
+ status = message->notmuch->mailstore->sync_tags (message->notmuch->mailstore,
+ message);
+ if (status != NOTMUCH_PRIVATE_STATUS_SUCCESS) {
+ fprintf (stderr, "Error: Cannot sync tags to mailstore (%s)\n",
+ notmuch_status_to_string ((notmuch_status_t)status));
+ /* Exit to avoid unsynchronized mailstore. */
+ exit(1);
+ }
+ }
+
db = static_cast <Xapian::WritableDatabase *> (message->notmuch->xapian_db);
db->replace_document (message->doc_id, message->doc);
}
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 0d9cb0f..4f4cc8b 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -32,6 +32,9 @@
NOTMUCH_BEGIN_DECLS
#include <time.h>
+#include <sys/time.h> /* For struct timeval */
+#include <signal.h> /* For sig_atomic_t */
+#include <stdio.h> /* For FILE */
#ifndef FALSE
#define FALSE 0
@@ -119,6 +122,8 @@ typedef struct _notmuch_message notmuch_message_t;
typedef struct _notmuch_tags notmuch_tags_t;
typedef struct _notmuch_directory notmuch_directory_t;
typedef struct _notmuch_filenames notmuch_filenames_t;
+typedef struct _notmuch_mailstore notmuch_mailstore_t;
+
/* Create a new, empty notmuch database located at 'path'.
*
@@ -127,6 +132,8 @@ typedef struct _notmuch_filenames notmuch_filenames_t;
* create a new ".notmuch" directory within 'path' where notmuch will
* store its data.
*
+ * FIXME: Mailstore...
+ *
* After a successful call to notmuch_database_create, the returned
* database will be open so the caller should call
* notmuch_database_close when finished with it.
@@ -140,7 +147,7 @@ typedef struct _notmuch_filenames notmuch_filenames_t;
* an error message on stderr).
*/
notmuch_database_t *
-notmuch_database_create (const char *path);
+notmuch_database_create (const char *path, notmuch_mailstore_t *mailstore);
typedef enum {
NOTMUCH_DATABASE_MODE_READ_ONLY = 0,
@@ -152,6 +159,8 @@ typedef enum {
/* Open an existing notmuch database located at 'path'.
*
+ * The database will index mails stored in 'mailstore'.
+ *
* The database should have been created at some time in the past,
* (not necessarily by this process), by calling
* notmuch_database_create with 'path'. By default the database should be
@@ -169,7 +178,8 @@ typedef enum {
*/
notmuch_database_t *
notmuch_database_open (const char *path,
- notmuch_database_mode_t mode);
+ notmuch_database_mode_t mode,
+ notmuch_mailstore_t *mailstore);
/* Close the given notmuch database, freeing all associated
* resources. See notmuch_database_open. */
@@ -236,7 +246,8 @@ notmuch_database_get_directory (notmuch_database_t *database,
* notmuch database will reference the filename, and will not copy the
* entire contents of the file.
*
- * If 'message' is not NULL, then, on successful return '*message'
+ * If 'message' is not NULL, then, on successful return
+ * (NOTMUCH_STATUS_SUCCESS or NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) '*message'
* will be initialized to a message object that can be used for things
* such as adding tags to the just-added message. The user should call
* notmuch_message_destroy when done with the message. On any failure
@@ -495,7 +506,7 @@ notmuch_threads_destroy (notmuch_threads_t *threads);
*/
unsigned
notmuch_query_count_messages (notmuch_query_t *query);
-
+
/* Get the thread ID of 'thread'.
*
* The returned string belongs to 'thread' and as such, should not be
@@ -1104,6 +1115,72 @@ notmuch_filenames_move_to_next (notmuch_filenames_t *filenames);
void
notmuch_filenames_destroy (notmuch_filenames_t *filenames);
+/* Returns notmuch_mailstore_t object of a given 'type'.
+ *
+ * The 'type' is case insensitive. If 'type' is NULL, a default
+ * mailstore is returned.
+ *
+ * When no available mailstore matches 'type', NULL is returned.
+ */
+notmuch_mailstore_t *
+notmuch_mailstore_get_by_type (const char *type);
+
+/* Returns the type of a mailstore.
+ *
+ * The returned string is owned by mailstore so should not be modified
+ * nor freed by the caller.
+ */
+const char *
+notmuch_mailstore_get_type (notmuch_mailstore_t *mailstore);
+
+/* Count emails in mailstore.
+ *
+ * This function returns when interrupted is set to non-zero value.
+ */
+void
+notmuch_mailstore_count_files (notmuch_mailstore_t *mailstore,
+ const char *path, int *count,
+ volatile sig_atomic_t *interrupted);
+
+typedef struct notmuch_indexing_context {
+ int output_is_a_tty;
+ int verbose;
+
+ int total_files;
+ int processed_files;
+ int added_messages;
+ int renamed_files;
+ int removed_files;
+ struct timeval tv_start;
+
+ void *priv;
+
+ volatile sig_atomic_t interrupted;
+ volatile sig_atomic_t print_progress;
+ void (*print_progress_cb)(struct notmuch_indexing_context *ctx);
+
+} notmuch_indexing_context_t;
+
+
+/* Index new mails in the mailstore.
+ *
+ * Mailstore is responsible for detecting new emails, adding them to
+ * the database and tagging them with tags stored in the mailstore.
+ */
+notmuch_status_t
+notmuch_mailstore_index_new (notmuch_mailstore_t *mailstore,
+ const char *path,
+ notmuch_indexing_context_t *ctx);
+
+/* Returns file handle to the file in the mailstore.
+ *
+ * TODO: This function is currently not used and access to mails is
+ * hardcoded in other parts of notmuch.
+ */
+FILE *
+notmuch_mailstore_open_file (notmuch_mailstore_t *mailstore,
+ const char *filename);
+
NOTMUCH_END_DECLS
#endif
diff --git a/notmuch-client.h b/notmuch-client.h
index c80b39c..4794b8e 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -171,6 +171,13 @@ notmuch_config_set_user_other_email (notmuch_config_t *config,
const char *other_email[],
size_t length);
+notmuch_mailstore_t *
+notmuch_config_get_mailstore(notmuch_config_t *config);
+
+void
+notmuch_config_set_mailstore(notmuch_config_t *config,
+ notmuch_mailstore_t *mailstore);
+
notmuch_bool_t
debugger_is_active (void);
diff --git a/notmuch-config.c b/notmuch-config.c
index 95430db..299423a 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -62,6 +62,7 @@ struct _notmuch_config {
char *user_primary_email;
char **user_other_email;
size_t user_other_email_length;
+ notmuch_mailstore_t *mailstore;
};
static int
@@ -199,6 +200,7 @@ notmuch_config_open (void *ctx,
config->user_primary_email = NULL;
config->user_other_email = NULL;
config->user_other_email_length = 0;
+ config->mailstore = NULL;
if (! g_key_file_load_from_file (config->key_file,
config->filename,
@@ -264,6 +266,10 @@ notmuch_config_open (void *ctx,
}
}
+ if (notmuch_config_get_mailstore (config) == NULL) {
+ notmuch_config_set_mailstore(config, notmuch_mailstore_get_by_type(NULL));
+ }
+
/* When we create a new configuration file here, we add some
* comments to help the user understand what can be done. */
if (is_new) {
@@ -452,3 +458,31 @@ notmuch_config_set_user_other_email (notmuch_config_t *config,
talloc_free (config->user_other_email);
config->user_other_email = NULL;
}
+
+notmuch_mailstore_t *
+notmuch_config_get_mailstore(notmuch_config_t *config)
+{
+ char *type;
+
+ if (config->mailstore == NULL) {
+ type = g_key_file_get_string (config->key_file,
+ "mailstore", "type", NULL);
+ if (type) {
+ config->mailstore = notmuch_mailstore_get_by_type(type);
+ free (type);
+ }
+ }
+
+ return config->mailstore;
+}
+
+void
+notmuch_config_set_mailstore(notmuch_config_t *config,
+ notmuch_mailstore_t *mailstore)
+{
+ g_key_file_set_string (config->key_file,
+ "mailstore", "type",
+ notmuch_mailstore_get_type(mailstore));
+
+ config->mailstore = NULL;
+}
diff --git a/notmuch-count.c b/notmuch-count.c
index 77aa433..631b8f8 100644
--- a/notmuch-count.c
+++ b/notmuch-count.c
@@ -81,7 +81,8 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
return 1;
notmuch = notmuch_database_open (notmuch_config_get_database_path (config),
- NOTMUCH_DATABASE_MODE_READ_ONLY);
+ NOTMUCH_DATABASE_MODE_READ_ONLY,
+ notmuch_config_get_mailstore(config));
if (notmuch == NULL)
return 1;
diff --git a/notmuch-dump.c b/notmuch-dump.c
index 7e7bc17..e74dfcb 100644
--- a/notmuch-dump.c
+++ b/notmuch-dump.c
@@ -36,7 +36,8 @@ notmuch_dump_command (unused (void *ctx), int argc, char *argv[])
return 1;
notmuch = notmuch_database_open (notmuch_config_get_database_path (config),
- NOTMUCH_DATABASE_MODE_READ_ONLY);
+ NOTMUCH_DATABASE_MODE_READ_ONLY,
+ notmuch_config_get_mailstore (config));
if (notmuch == NULL)
return 1;
diff --git a/notmuch-new.c b/notmuch-new.c
index 44b50aa..2d0ba6c 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -749,11 +749,13 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
return 1;
printf ("Found %d total files (that's not much mail).\n", count);
- notmuch = notmuch_database_create (db_path);
+ notmuch = notmuch_database_create (db_path,
+ notmuch_config_get_mailstore (config));
add_files_state.total_files = count;
} else {
notmuch = notmuch_database_open (db_path,
- NOTMUCH_DATABASE_MODE_READ_WRITE);
+ NOTMUCH_DATABASE_MODE_READ_WRITE,
+ notmuch_config_get_mailstore (config));
if (notmuch == NULL)
return 1;
diff --git a/notmuch-reply.c b/notmuch-reply.c
index 6c15536..0d6b7cf 100644
--- a/notmuch-reply.c
+++ b/notmuch-reply.c
@@ -468,7 +468,8 @@ notmuch_reply_command (void *ctx, int argc, char *argv[])
}
notmuch = notmuch_database_open (notmuch_config_get_database_path (config),
- NOTMUCH_DATABASE_MODE_READ_ONLY);
+ NOTMUCH_DATABASE_MODE_READ_ONLY,
+ notmuch_config_get_mailstore (config));
if (notmuch == NULL)
return 1;
diff --git a/notmuch-restore.c b/notmuch-restore.c
index b0a4e1c..97d817b 100644
--- a/notmuch-restore.c
+++ b/notmuch-restore.c
@@ -37,7 +37,8 @@ notmuch_restore_command (unused (void *ctx), int argc, char *argv[])
return 1;
notmuch = notmuch_database_open (notmuch_config_get_database_path (config),
- NOTMUCH_DATABASE_MODE_READ_WRITE);
+ NOTMUCH_DATABASE_MODE_READ_WRITE,
+ notmuch_config_get_mailstore (config));
if (notmuch == NULL)
return 1;
diff --git a/notmuch-search-tags.c b/notmuch-search-tags.c
index 6f3cfcc..323f416 100644
--- a/notmuch-search-tags.c
+++ b/notmuch-search-tags.c
@@ -52,7 +52,8 @@ notmuch_search_tags_command (void *ctx, int argc, char *argv[])
}
db = notmuch_database_open (notmuch_config_get_database_path (config),
- NOTMUCH_DATABASE_MODE_READ_ONLY);
+ NOTMUCH_DATABASE_MODE_READ_ONLY,
+ notmuch_config_get_mailstore (config));
if (db == NULL) {
goto error;
}
diff --git a/notmuch-search.c b/notmuch-search.c
index 4e3514b..d57479d 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -245,7 +245,8 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
return 1;
notmuch = notmuch_database_open (notmuch_config_get_database_path (config),
- NOTMUCH_DATABASE_MODE_READ_ONLY);
+ NOTMUCH_DATABASE_MODE_READ_ONLY,
+ notmuch_config_get_mailstore (config));
if (notmuch == NULL)
return 1;
diff --git a/notmuch-show.c b/notmuch-show.c
index ff1fecb..26581c1 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -458,7 +458,8 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
}
notmuch = notmuch_database_open (notmuch_config_get_database_path (config),
- NOTMUCH_DATABASE_MODE_READ_ONLY);
+ NOTMUCH_DATABASE_MODE_READ_ONLY,
+ notmuch_config_get_mailstore (config));
if (notmuch == NULL)
return 1;
diff --git a/notmuch-tag.c b/notmuch-tag.c
index 8b6f7dc..5e1a5c3 100644
--- a/notmuch-tag.c
+++ b/notmuch-tag.c
@@ -97,7 +97,8 @@ notmuch_tag_command (void *ctx, unused (int argc), unused (char *argv[]))
return 1;
notmuch = notmuch_database_open (notmuch_config_get_database_path (config),
- NOTMUCH_DATABASE_MODE_READ_WRITE);
+ NOTMUCH_DATABASE_MODE_READ_WRITE,
+ notmuch_config_get_mailstore (config));
if (notmuch == NULL)
return 1;
--
1.7.0
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [PATCH 2/4] Convert mailstore abstraction
2010-03-18 15:39 Mailstore abstraction & maildir synchronization Michal Sojka
2010-03-18 15:39 ` [PATCH 1/4] Mailstore abstraction interface Michal Sojka
@ 2010-03-18 15:39 ` Michal Sojka
2010-03-18 15:39 ` [PATCH 3/4] Add maildir-based mailstore Michal Sojka
` (2 subsequent siblings)
4 siblings, 0 replies; 25+ messages in thread
From: Michal Sojka @ 2010-03-18 15:39 UTC (permalink / raw)
To: notmuch
The code for detection of new files in the mailstore and their
addition to the database is moved from notmuch-new.c to
lib/mailstore-files.c, where it is called by the abstract mailstore
interface.
The code was changed to allow the progress reporting function to be
implemented outside of notmuch library.
Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz>
---
lib/mailstore-files.c | 590 +++++++++++++++++++++++++++++++++++++++++++++++
notmuch-new.c | 611 ++-----------------------------------------------
2 files changed, 615 insertions(+), 586 deletions(-)
diff --git a/lib/mailstore-files.c b/lib/mailstore-files.c
index 92d7f5d..ace2664 100644
--- a/lib/mailstore-files.c
+++ b/lib/mailstore-files.c
@@ -20,9 +20,596 @@
* Michal Sojka <sojkam1@fel.cvut.cz>
*/
+#define _GNU_SOURCE /* For asprintf() */
#include "notmuch.h"
#include "mailstore-private.h"
+#include <dirent.h>
+typedef struct _filename_node {
+ char *filename;
+ struct _filename_node *next;
+} _filename_node_t;
+
+typedef struct _filename_list {
+ _filename_node_t *head;
+ _filename_node_t **tail;
+} _filename_list_t;
+
+typedef struct _indexing_context_priv {
+ _filename_list_t *removed_files;
+ _filename_list_t *removed_directories;
+} _indexing_context_priv_t;
+
+static _filename_list_t *
+_filename_list_create (const void *ctx)
+{
+ _filename_list_t *list;
+
+ list = talloc (ctx, _filename_list_t);
+ if (list == NULL)
+ return NULL;
+
+ list->head = NULL;
+ list->tail = &list->head;
+
+ return list;
+}
+
+static void
+_filename_list_add (_filename_list_t *list,
+ const char *filename)
+{
+ _filename_node_t *node = talloc (list, _filename_node_t);
+
+ node->filename = talloc_strdup (list, filename);
+ node->next = NULL;
+
+ *(list->tail) = node;
+ list->tail = &node->next;
+}
+
+static void
+tag_inbox_and_unread (notmuch_message_t *message)
+{
+ notmuch_message_add_tag (message, "inbox");
+ notmuch_message_add_tag (message, "unread");
+}
+
+static int
+dirent_sort_inode (const struct dirent **a, const struct dirent **b)
+{
+ return ((*a)->d_ino < (*b)->d_ino) ? -1 : 1;
+}
+
+static int
+dirent_sort_strcmp_name (const struct dirent **a, const struct dirent **b)
+{
+ return strcmp ((*a)->d_name, (*b)->d_name);
+}
+
+/* Test if the directory looks like a Maildir directory.
+ *
+ * Search through the array of directory entries to see if we can find all
+ * three subdirectories typical for Maildir, that is "new", "cur", and "tmp".
+ *
+ * Return 1 if the directory looks like a Maildir and 0 otherwise.
+ */
+static int
+_entries_resemble_maildir (struct dirent **entries, int count)
+{
+ int i, found = 0;
+
+ for (i = 0; i < count; i++) {
+ if (entries[i]->d_type != DT_DIR && entries[i]->d_type != DT_UNKNOWN)
+ continue;
+
+ if (strcmp(entries[i]->d_name, "new") == 0 ||
+ strcmp(entries[i]->d_name, "cur") == 0 ||
+ strcmp(entries[i]->d_name, "tmp") == 0)
+ {
+ found++;
+ if (found == 3)
+ return 1;
+ }
+ }
+
+ return 0;
+}
+
+
+/* Examine 'path' recursively as follows:
+ *
+ * o Ask the filesystem for the mtime of 'path' (fs_mtime)
+ * o Ask the database for its timestamp of 'path' (db_mtime)
+ *
+ * o Ask the filesystem for files and directories within 'path'
+ * (via scandir and stored in fs_entries)
+ * o Ask the database for files and directories within 'path'
+ * (db_files and db_subdirs)
+ *
+ * o Pass 1: For each directory in fs_entries, recursively call into
+ * this same function.
+ *
+ * o Pass 2: If 'fs_mtime' > 'db_mtime', then walk fs_entries
+ * simultaneously with db_files and db_subdirs. Look for one of
+ * three interesting cases:
+ *
+ * 1. Regular file in fs_entries and not in db_files
+ * This is a new file to add_message into the database.
+ *
+ * 2. Filename in db_files not in fs_entries.
+ * This is a file that has been removed from the mail store.
+ *
+ * 3. Directory in db_subdirs not in fs_entries
+ * This is a directory that has been removed from the mail store.
+ *
+ * Note that the addition of a directory is not interesting here,
+ * since that will have been taken care of in pass 1. Also, we
+ * don't immediately act on file/directory removal since we must
+ * ensure that in the case of a rename that the new filename is
+ * added before the old filename is removed, (so that no
+ * information is lost from the database).
+ *
+ * o Tell the database to update its time of 'path' to 'fs_mtime'
+ */
+static notmuch_status_t
+add_files_recursive (notmuch_mailstore_t *mailstore,
+ const char *path,
+ notmuch_indexing_context_t *state)
+{
+ DIR *dir = NULL;
+ struct dirent *entry = NULL;
+ char *next = NULL;
+ time_t fs_mtime, db_mtime;
+ notmuch_status_t status, ret = NOTMUCH_STATUS_SUCCESS;
+ notmuch_message_t *message = NULL;
+ struct dirent **fs_entries = NULL;
+ int i, num_fs_entries;
+ notmuch_directory_t *directory;
+ notmuch_filenames_t *db_files = NULL;
+ notmuch_filenames_t *db_subdirs = NULL;
+ struct stat st;
+ notmuch_bool_t is_maildir, new_directory;
+ _indexing_context_priv_t *priv = state->priv;
+ notmuch_database_t *notmuch = mailstore->notmuch;
+
+ if (stat (path, &st)) {
+ fprintf (stderr, "Error reading directory %s: %s\n",
+ path, strerror (errno));
+ return NOTMUCH_STATUS_FILE_ERROR;
+ }
+
+ /* This is not an error since we may have recursed based on a
+ * symlink to a regular file, not a directory, and we don't know
+ * that until this stat. */
+ if (! S_ISDIR (st.st_mode))
+ return NOTMUCH_STATUS_SUCCESS;
+
+ fs_mtime = st.st_mtime;
+
+ directory = notmuch_database_get_directory (notmuch, path);
+ db_mtime = notmuch_directory_get_mtime (directory);
+
+ if (db_mtime == 0) {
+ new_directory = TRUE;
+ db_files = NULL;
+ db_subdirs = NULL;
+ } else {
+ new_directory = FALSE;
+ db_files = notmuch_directory_get_child_files (directory);
+ db_subdirs = notmuch_directory_get_child_directories (directory);
+ }
+
+ /* If the database knows about this directory, then we sort based
+ * on strcmp to match the database sorting. Otherwise, we can do
+ * inode-based sorting for faster filesystem operation. */
+ num_fs_entries = scandir (path, &fs_entries, 0,
+ new_directory ?
+ dirent_sort_inode : dirent_sort_strcmp_name);
+
+ if (num_fs_entries == -1) {
+ fprintf (stderr, "Error opening directory %s: %s\n",
+ path, strerror (errno));
+ ret = NOTMUCH_STATUS_FILE_ERROR;
+ goto DONE;
+ }
+
+ /* Pass 1: Recurse into all sub-directories. */
+ is_maildir = _entries_resemble_maildir (fs_entries, num_fs_entries);
+
+ for (i = 0; i < num_fs_entries; i++) {
+ if (state->interrupted)
+ break;
+
+ entry = fs_entries[i];
+
+ /* We only want to descend into directories.
+ * But symlinks can be to directories too, of course.
+ *
+ * And if the filesystem doesn't tell us the file type in the
+ * scandir results, then it might be a directory (and if not,
+ * then we'll stat and return immediately in the next level of
+ * recursion). */
+ if (entry->d_type != DT_DIR &&
+ entry->d_type != DT_LNK &&
+ entry->d_type != DT_UNKNOWN)
+ {
+ continue;
+ }
+
+ /* Ignore special directories to avoid infinite recursion.
+ * Also ignore the .notmuch directory and any "tmp" directory
+ * that appears within a maildir.
+ */
+ /* XXX: Eventually we'll want more sophistication to let the
+ * user specify files to be ignored. */
+ if (strcmp (entry->d_name, ".") == 0 ||
+ strcmp (entry->d_name, "..") == 0 ||
+ (is_maildir && strcmp (entry->d_name, "tmp") == 0) ||
+ strcmp (entry->d_name, ".notmuch") ==0)
+ {
+ continue;
+ }
+
+ next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
+ status = add_files_recursive (mailstore, next, state);
+ if (status && ret == NOTMUCH_STATUS_SUCCESS)
+ ret = status;
+ talloc_free (next);
+ next = NULL;
+ }
+
+ /* If this directory hasn't been modified since the last
+ * "notmuch new", then we can skip the second pass entirely. */
+ if (fs_mtime <= db_mtime)
+ goto DONE;
+
+ /* Pass 2: Scan for new files, removed files, and removed directories. */
+ for (i = 0; i < num_fs_entries; i++)
+ {
+ if (state->interrupted)
+ break;
+
+ entry = fs_entries[i];
+
+ /* Check if we've walked past any names in db_files or
+ * db_subdirs. If so, these have been deleted. */
+ while (notmuch_filenames_valid (db_files) &&
+ strcmp (notmuch_filenames_get (db_files), entry->d_name) < 0)
+ {
+ char *absolute = talloc_asprintf (priv->removed_files,
+ "%s/%s", path,
+ notmuch_filenames_get (db_files));
+
+ _filename_list_add (priv->removed_files, absolute);
+
+ notmuch_filenames_move_to_next (db_files);
+ }
+
+ while (notmuch_filenames_valid (db_subdirs) &&
+ strcmp (notmuch_filenames_get (db_subdirs), entry->d_name) <= 0)
+ {
+ const char *filename = notmuch_filenames_get (db_subdirs);
+
+ if (strcmp (filename, entry->d_name) < 0)
+ {
+ char *absolute = talloc_asprintf (priv->removed_directories,
+ "%s/%s", path, filename);
+
+ _filename_list_add (priv->removed_directories, absolute);
+ }
+
+ notmuch_filenames_move_to_next (db_subdirs);
+ }
+
+ /* If we're looking at a symlink, we only want to add it if it
+ * links to a regular file, (and not to a directory, say).
+ *
+ * Similarly, if the file is of unknown type (due to filesytem
+ * limitations), then we also need to look closer.
+ *
+ * In either case, a stat does the trick.
+ */
+ if (entry->d_type == DT_LNK || entry->d_type == DT_UNKNOWN) {
+ int err;
+
+ next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
+ err = stat (next, &st);
+ talloc_free (next);
+ next = NULL;
+
+ /* Don't emit an error for a link pointing nowhere, since
+ * the directory-traversal pass will have already done
+ * that. */
+ if (err)
+ continue;
+
+ if (! S_ISREG (st.st_mode))
+ continue;
+ } else if (entry->d_type != DT_REG) {
+ continue;
+ }
+
+ /* Don't add a file that we've added before. */
+ if (notmuch_filenames_valid (db_files) &&
+ strcmp (notmuch_filenames_get (db_files), entry->d_name) == 0)
+ {
+ notmuch_filenames_move_to_next (db_files);
+ continue;
+ }
+
+ /* We're now looking at a regular file that doesn't yet exist
+ * in the database, so add it. */
+ next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
+
+ state->processed_files++;
+
+ if (state->verbose) {
+ if (state->output_is_a_tty)
+ printf("\r\033[K");
+
+ printf ("%i/%i: %s",
+ state->processed_files,
+ state->total_files,
+ next);
+
+ putchar((state->output_is_a_tty) ? '\r' : '\n');
+ fflush (stdout);
+ }
+
+ status = notmuch_database_add_message (notmuch, next, &message);
+ switch (status) {
+ /* success */
+ case NOTMUCH_STATUS_SUCCESS:
+ state->added_messages++;
+ tag_inbox_and_unread (message);
+ break;
+ /* Non-fatal issues (go on to next file) */
+ case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
+ /* Stay silent on this one. */
+ break;
+ case NOTMUCH_STATUS_FILE_NOT_EMAIL:
+ fprintf (stderr, "Note: Ignoring non-mail file: %s\n",
+ next);
+ break;
+ /* Fatal issues. Don't process anymore. */
+ case NOTMUCH_STATUS_READ_ONLY_DATABASE:
+ case NOTMUCH_STATUS_XAPIAN_EXCEPTION:
+ case NOTMUCH_STATUS_OUT_OF_MEMORY:
+ fprintf (stderr, "Error: %s. Halting processing.\n",
+ notmuch_status_to_string (status));
+ ret = status;
+ goto DONE;
+ default:
+ case NOTMUCH_STATUS_FILE_ERROR:
+ case NOTMUCH_STATUS_NULL_POINTER:
+ case NOTMUCH_STATUS_TAG_TOO_LONG:
+ case NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW:
+ case NOTMUCH_STATUS_LAST_STATUS:
+ INTERNAL_ERROR ("add_message returned unexpected value: %d", status);
+ goto DONE;
+ }
+
+ if (message) {
+ notmuch_message_destroy (message);
+ message = NULL;
+ }
+
+ if (state->print_progress &&
+ state->print_progress_cb) {
+ state->print_progress = 0;
+ state->print_progress_cb (state);
+ }
+
+ talloc_free (next);
+ next = NULL;
+ }
+
+ /* FIXME: Handle interrupted - there might be data loss */
+
+ /* Now that we've walked the whole filesystem list, anything left
+ * over in the database lists has been deleted. */
+ while (notmuch_filenames_valid (db_files))
+ {
+ char *absolute = talloc_asprintf (priv->removed_files,
+ "%s/%s", path,
+ notmuch_filenames_get (db_files));
+
+ _filename_list_add (priv->removed_files, absolute);
+
+ notmuch_filenames_move_to_next (db_files);
+ }
+
+ while (notmuch_filenames_valid (db_subdirs))
+ {
+ char *absolute = talloc_asprintf (priv->removed_directories,
+ "%s/%s", path,
+ notmuch_filenames_get (db_subdirs));
+
+ _filename_list_add (priv->removed_directories, absolute);
+
+ notmuch_filenames_move_to_next (db_subdirs);
+ }
+
+ if (! state->interrupted) {
+ status = notmuch_directory_set_mtime (directory, fs_mtime);
+ if (status && ret == NOTMUCH_STATUS_SUCCESS)
+ ret = status;
+ }
+
+DONE:
+ if (next)
+ talloc_free (next);
+ if (entry)
+ free (entry);
+ if (dir)
+ closedir (dir);
+ if (fs_entries)
+ free (fs_entries);
+ if (db_subdirs)
+ notmuch_filenames_destroy (db_subdirs);
+ if (db_files)
+ notmuch_filenames_destroy (db_files);
+ if (directory)
+ notmuch_directory_destroy (directory);
+
+ return ret;
+}
+
+/* XXX: This should be merged with the add_files function since it
+ * shares a lot of logic with it. */
+/* Recursively count all regular files in path and all sub-directories
+ * of path. The result is added to *count (which should be
+ * initialized to zero by the top-level caller before calling
+ * count_files). */
+static void
+count_files (notmuch_mailstore_t *mailstore,
+ const char *path, int *count,
+ volatile sig_atomic_t *interrupted)
+{
+ struct dirent *entry = NULL;
+ char *next;
+ struct stat st;
+ struct dirent **fs_entries = NULL;
+ int num_fs_entries = scandir (path, &fs_entries, 0, dirent_sort_inode);
+ int i = 0;
+
+ (void)mailstore;
+ if (num_fs_entries == -1) {
+ fprintf (stderr, "Warning: failed to open directory %s: %s\n",
+ path, strerror (errno));
+ goto DONE;
+ }
+
+ while (!*interrupted) {
+ if (i == num_fs_entries)
+ break;
+
+ entry = fs_entries[i++];
+
+ /* Ignore special directories to avoid infinite recursion.
+ * Also ignore the .notmuch directory.
+ */
+ /* XXX: Eventually we'll want more sophistication to let the
+ * user specify files to be ignored. */
+ if (strcmp (entry->d_name, ".") == 0 ||
+ strcmp (entry->d_name, "..") == 0 ||
+ strcmp (entry->d_name, ".notmuch") == 0)
+ {
+ continue;
+ }
+
+ if (asprintf (&next, "%s/%s", path, entry->d_name) == -1) {
+ next = NULL;
+ fprintf (stderr, "Error descending from %s to %s: Out of memory\n",
+ path, entry->d_name);
+ continue;
+ }
+
+ stat (next, &st);
+
+ if (S_ISREG (st.st_mode)) {
+ *count = *count + 1;
+ if (*count % 1000 == 0) {
+ printf ("Found %d files so far.\r", *count);
+ fflush (stdout);
+ }
+ } else if (S_ISDIR (st.st_mode)) {
+ count_files (mailstore, next, count, interrupted);
+ }
+
+ free (next);
+ }
+
+DONE:
+ if (entry)
+ free (entry);
+ if (fs_entries)
+ free (fs_entries);
+}
+
+/* Recursively remove all filenames from the database referring to
+ * 'path' (or to any of its children). */
+static void
+_remove_directory (void *ctx,
+ notmuch_database_t *notmuch,
+ const char *path,
+ int *renamed_files,
+ int *removed_files)
+{
+ notmuch_directory_t *directory;
+ notmuch_filenames_t *files, *subdirs;
+ notmuch_status_t status;
+ char *absolute;
+
+ directory = notmuch_database_get_directory (notmuch, path);
+
+ for (files = notmuch_directory_get_child_files (directory);
+ notmuch_filenames_valid (files);
+ notmuch_filenames_move_to_next (files))
+ {
+ absolute = talloc_asprintf (ctx, "%s/%s", path,
+ notmuch_filenames_get (files));
+ status = notmuch_database_remove_message (notmuch, absolute);
+ if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID)
+ *renamed_files = *renamed_files + 1;
+ else
+ *removed_files = *removed_files + 1;
+ talloc_free (absolute);
+ }
+
+ for (subdirs = notmuch_directory_get_child_directories (directory);
+ notmuch_filenames_valid (subdirs);
+ notmuch_filenames_move_to_next (subdirs))
+ {
+ absolute = talloc_asprintf (ctx, "%s/%s", path,
+ notmuch_filenames_get (subdirs));
+ _remove_directory (ctx, notmuch, absolute, renamed_files, removed_files);
+ talloc_free (absolute);
+ }
+
+ notmuch_directory_destroy (directory);
+}
+
+static notmuch_private_status_t
+index_new(notmuch_mailstore_t *mailstore, const char* path,
+ notmuch_indexing_context_t *indexing_ctx)
+{
+ _indexing_context_priv_t *priv;
+ _filename_node_t *f;
+ notmuch_status_t status, ret;
+ notmuch_database_t *notmuch = mailstore->notmuch;
+
+ priv = talloc(NULL, _indexing_context_priv_t);
+ indexing_ctx->priv = priv;
+ if (priv == NULL)
+ return NOTMUCH_STATUS_OUT_OF_MEMORY;
+
+ priv->removed_files = _filename_list_create (priv);
+ priv->removed_directories = _filename_list_create (priv);
+
+ ret = add_files_recursive(mailstore, path, indexing_ctx);
+
+ indexing_ctx->removed_files = 0;
+ indexing_ctx->renamed_files = 0;
+ for (f = priv->removed_files->head; f; f = f->next) {
+ status = notmuch_database_remove_message (notmuch, f->filename);
+ if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID)
+ indexing_ctx->renamed_files++;
+ else
+ indexing_ctx->removed_files++;
+ }
+
+ for (f = priv->removed_directories->head; f; f = f->next) {
+ _remove_directory (priv, notmuch, f->filename,
+ &indexing_ctx->renamed_files,
+ &indexing_ctx->removed_files);
+ }
+
+ talloc_free(priv);
+
+ return ret;
+}
static FILE *
open_file(notmuch_mailstore_t *mailstore, const char *filename)
@@ -39,5 +626,8 @@ open_file(notmuch_mailstore_t *mailstore, const char *filename)
/* Original notmuch mail store */
struct _notmuch_mailstore mailstore_files = {
.type = "files",
+ .count_files = count_files,
+ .index_new = index_new,
+ .sync_tags = NULL,
.open_file = open_file,
};
diff --git a/notmuch-new.c b/notmuch-new.c
index 2d0ba6c..ede0859 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -22,35 +22,12 @@
#include <unistd.h>
-typedef struct _filename_node {
- char *filename;
- struct _filename_node *next;
-} _filename_node_t;
-
-typedef struct _filename_list {
- _filename_node_t *head;
- _filename_node_t **tail;
-} _filename_list_t;
-
-typedef struct {
- int output_is_a_tty;
- int verbose;
-
- int total_files;
- int processed_files;
- int added_messages;
- struct timeval tv_start;
-
- _filename_list_t *removed_files;
- _filename_list_t *removed_directories;
-} add_files_state_t;
-
-static volatile sig_atomic_t do_add_files_print_progress = 0;
+notmuch_indexing_context_t *indexing_ctx;
static void
handle_sigalrm (unused (int signal))
{
- do_add_files_print_progress = 1;
+ indexing_ctx->print_progress = 1;
}
static volatile sig_atomic_t interrupted;
@@ -62,46 +39,11 @@ handle_sigint (unused (int sig))
static char msg[] = "Stopping... \n";
ignored = write(2, msg, sizeof(msg)-1);
- interrupted = 1;
-}
-
-static _filename_list_t *
-_filename_list_create (const void *ctx)
-{
- _filename_list_t *list;
-
- list = talloc (ctx, _filename_list_t);
- if (list == NULL)
- return NULL;
-
- list->head = NULL;
- list->tail = &list->head;
-
- return list;
-}
-
-static void
-_filename_list_add (_filename_list_t *list,
- const char *filename)
-{
- _filename_node_t *node = talloc (list, _filename_node_t);
-
- node->filename = talloc_strdup (list, filename);
- node->next = NULL;
-
- *(list->tail) = node;
- list->tail = &node->next;
+ indexing_ctx->interrupted = 1;
}
static void
-tag_inbox_and_unread (notmuch_message_t *message)
-{
- notmuch_message_add_tag (message, "inbox");
- notmuch_message_add_tag (message, "unread");
-}
-
-static void
-add_files_print_progress (add_files_state_t *state)
+add_files_print_progress (notmuch_indexing_context_t *state)
{
struct timeval tv_now;
double elapsed_overall, rate_overall;
@@ -128,388 +70,13 @@ add_files_print_progress (add_files_state_t *state)
fflush (stdout);
}
-static int
-dirent_sort_inode (const struct dirent **a, const struct dirent **b)
-{
- return ((*a)->d_ino < (*b)->d_ino) ? -1 : 1;
-}
-
-static int
-dirent_sort_strcmp_name (const struct dirent **a, const struct dirent **b)
-{
- return strcmp ((*a)->d_name, (*b)->d_name);
-}
-
-/* Test if the directory looks like a Maildir directory.
- *
- * Search through the array of directory entries to see if we can find all
- * three subdirectories typical for Maildir, that is "new", "cur", and "tmp".
- *
- * Return 1 if the directory looks like a Maildir and 0 otherwise.
- */
-static int
-_entries_resemble_maildir (struct dirent **entries, int count)
-{
- int i, found = 0;
-
- for (i = 0; i < count; i++) {
- if (entries[i]->d_type != DT_DIR && entries[i]->d_type != DT_UNKNOWN)
- continue;
-
- if (strcmp(entries[i]->d_name, "new") == 0 ||
- strcmp(entries[i]->d_name, "cur") == 0 ||
- strcmp(entries[i]->d_name, "tmp") == 0)
- {
- found++;
- if (found == 3)
- return 1;
- }
- }
-
- return 0;
-}
-
-/* Examine 'path' recursively as follows:
- *
- * o Ask the filesystem for the mtime of 'path' (fs_mtime)
- * o Ask the database for its timestamp of 'path' (db_mtime)
- *
- * o Ask the filesystem for files and directories within 'path'
- * (via scandir and stored in fs_entries)
- * o Ask the database for files and directories within 'path'
- * (db_files and db_subdirs)
- *
- * o Pass 1: For each directory in fs_entries, recursively call into
- * this same function.
- *
- * o Pass 2: If 'fs_mtime' > 'db_mtime', then walk fs_entries
- * simultaneously with db_files and db_subdirs. Look for one of
- * three interesting cases:
- *
- * 1. Regular file in fs_entries and not in db_files
- * This is a new file to add_message into the database.
- *
- * 2. Filename in db_files not in fs_entries.
- * This is a file that has been removed from the mail store.
- *
- * 3. Directory in db_subdirs not in fs_entries
- * This is a directory that has been removed from the mail store.
- *
- * Note that the addition of a directory is not interesting here,
- * since that will have been taken care of in pass 1. Also, we
- * don't immediately act on file/directory removal since we must
- * ensure that in the case of a rename that the new filename is
- * added before the old filename is removed, (so that no
- * information is lost from the database).
- *
- * o Tell the database to update its time of 'path' to 'fs_mtime'
- */
-static notmuch_status_t
-add_files_recursive (notmuch_database_t *notmuch,
- const char *path,
- add_files_state_t *state)
-{
- DIR *dir = NULL;
- struct dirent *entry = NULL;
- char *next = NULL;
- time_t fs_mtime, db_mtime;
- notmuch_status_t status, ret = NOTMUCH_STATUS_SUCCESS;
- notmuch_message_t *message = NULL;
- struct dirent **fs_entries = NULL;
- int i, num_fs_entries;
- notmuch_directory_t *directory;
- notmuch_filenames_t *db_files = NULL;
- notmuch_filenames_t *db_subdirs = NULL;
- struct stat st;
- notmuch_bool_t is_maildir, new_directory;
-
- if (stat (path, &st)) {
- fprintf (stderr, "Error reading directory %s: %s\n",
- path, strerror (errno));
- return NOTMUCH_STATUS_FILE_ERROR;
- }
-
- /* This is not an error since we may have recursed based on a
- * symlink to a regular file, not a directory, and we don't know
- * that until this stat. */
- if (! S_ISDIR (st.st_mode))
- return NOTMUCH_STATUS_SUCCESS;
-
- fs_mtime = st.st_mtime;
-
- directory = notmuch_database_get_directory (notmuch, path);
- db_mtime = notmuch_directory_get_mtime (directory);
-
- if (db_mtime == 0) {
- new_directory = TRUE;
- db_files = NULL;
- db_subdirs = NULL;
- } else {
- new_directory = FALSE;
- db_files = notmuch_directory_get_child_files (directory);
- db_subdirs = notmuch_directory_get_child_directories (directory);
- }
-
- /* If the database knows about this directory, then we sort based
- * on strcmp to match the database sorting. Otherwise, we can do
- * inode-based sorting for faster filesystem operation. */
- num_fs_entries = scandir (path, &fs_entries, 0,
- new_directory ?
- dirent_sort_inode : dirent_sort_strcmp_name);
-
- if (num_fs_entries == -1) {
- fprintf (stderr, "Error opening directory %s: %s\n",
- path, strerror (errno));
- ret = NOTMUCH_STATUS_FILE_ERROR;
- goto DONE;
- }
-
- /* Pass 1: Recurse into all sub-directories. */
- is_maildir = _entries_resemble_maildir (fs_entries, num_fs_entries);
-
- for (i = 0; i < num_fs_entries; i++) {
- if (interrupted)
- break;
-
- entry = fs_entries[i];
-
- /* We only want to descend into directories.
- * But symlinks can be to directories too, of course.
- *
- * And if the filesystem doesn't tell us the file type in the
- * scandir results, then it might be a directory (and if not,
- * then we'll stat and return immediately in the next level of
- * recursion). */
- if (entry->d_type != DT_DIR &&
- entry->d_type != DT_LNK &&
- entry->d_type != DT_UNKNOWN)
- {
- continue;
- }
-
- /* Ignore special directories to avoid infinite recursion.
- * Also ignore the .notmuch directory and any "tmp" directory
- * that appears within a maildir.
- */
- /* XXX: Eventually we'll want more sophistication to let the
- * user specify files to be ignored. */
- if (strcmp (entry->d_name, ".") == 0 ||
- strcmp (entry->d_name, "..") == 0 ||
- (is_maildir && strcmp (entry->d_name, "tmp") == 0) ||
- strcmp (entry->d_name, ".notmuch") ==0)
- {
- continue;
- }
-
- next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
- status = add_files_recursive (notmuch, next, state);
- if (status && ret == NOTMUCH_STATUS_SUCCESS)
- ret = status;
- talloc_free (next);
- next = NULL;
- }
-
- /* If this directory hasn't been modified since the last
- * "notmuch new", then we can skip the second pass entirely. */
- if (fs_mtime <= db_mtime)
- goto DONE;
-
- /* Pass 2: Scan for new files, removed files, and removed directories. */
- for (i = 0; i < num_fs_entries; i++)
- {
- if (interrupted)
- break;
-
- entry = fs_entries[i];
-
- /* Check if we've walked past any names in db_files or
- * db_subdirs. If so, these have been deleted. */
- while (notmuch_filenames_valid (db_files) &&
- strcmp (notmuch_filenames_get (db_files), entry->d_name) < 0)
- {
- char *absolute = talloc_asprintf (state->removed_files,
- "%s/%s", path,
- notmuch_filenames_get (db_files));
-
- _filename_list_add (state->removed_files, absolute);
-
- notmuch_filenames_move_to_next (db_files);
- }
-
- while (notmuch_filenames_valid (db_subdirs) &&
- strcmp (notmuch_filenames_get (db_subdirs), entry->d_name) <= 0)
- {
- const char *filename = notmuch_filenames_get (db_subdirs);
-
- if (strcmp (filename, entry->d_name) < 0)
- {
- char *absolute = talloc_asprintf (state->removed_directories,
- "%s/%s", path, filename);
-
- _filename_list_add (state->removed_directories, absolute);
- }
-
- notmuch_filenames_move_to_next (db_subdirs);
- }
-
- /* If we're looking at a symlink, we only want to add it if it
- * links to a regular file, (and not to a directory, say).
- *
- * Similarly, if the file is of unknown type (due to filesytem
- * limitations), then we also need to look closer.
- *
- * In either case, a stat does the trick.
- */
- if (entry->d_type == DT_LNK || entry->d_type == DT_UNKNOWN) {
- int err;
-
- next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
- err = stat (next, &st);
- talloc_free (next);
- next = NULL;
-
- /* Don't emit an error for a link pointing nowhere, since
- * the directory-traversal pass will have already done
- * that. */
- if (err)
- continue;
-
- if (! S_ISREG (st.st_mode))
- continue;
- } else if (entry->d_type != DT_REG) {
- continue;
- }
-
- /* Don't add a file that we've added before. */
- if (notmuch_filenames_valid (db_files) &&
- strcmp (notmuch_filenames_get (db_files), entry->d_name) == 0)
- {
- notmuch_filenames_move_to_next (db_files);
- continue;
- }
-
- /* We're now looking at a regular file that doesn't yet exist
- * in the database, so add it. */
- next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
-
- state->processed_files++;
-
- if (state->verbose) {
- if (state->output_is_a_tty)
- printf("\r\033[K");
-
- printf ("%i/%i: %s",
- state->processed_files,
- state->total_files,
- next);
-
- putchar((state->output_is_a_tty) ? '\r' : '\n');
- fflush (stdout);
- }
-
- status = notmuch_database_add_message (notmuch, next, &message);
- switch (status) {
- /* success */
- case NOTMUCH_STATUS_SUCCESS:
- state->added_messages++;
- tag_inbox_and_unread (message);
- break;
- /* Non-fatal issues (go on to next file) */
- case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
- /* Stay silent on this one. */
- break;
- case NOTMUCH_STATUS_FILE_NOT_EMAIL:
- fprintf (stderr, "Note: Ignoring non-mail file: %s\n",
- next);
- break;
- /* Fatal issues. Don't process anymore. */
- case NOTMUCH_STATUS_READ_ONLY_DATABASE:
- case NOTMUCH_STATUS_XAPIAN_EXCEPTION:
- case NOTMUCH_STATUS_OUT_OF_MEMORY:
- fprintf (stderr, "Error: %s. Halting processing.\n",
- notmuch_status_to_string (status));
- ret = status;
- goto DONE;
- default:
- case NOTMUCH_STATUS_FILE_ERROR:
- case NOTMUCH_STATUS_NULL_POINTER:
- case NOTMUCH_STATUS_TAG_TOO_LONG:
- case NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW:
- case NOTMUCH_STATUS_LAST_STATUS:
- INTERNAL_ERROR ("add_message returned unexpected value: %d", status);
- goto DONE;
- }
-
- if (message) {
- notmuch_message_destroy (message);
- message = NULL;
- }
-
- if (do_add_files_print_progress) {
- do_add_files_print_progress = 0;
- add_files_print_progress (state);
- }
-
- talloc_free (next);
- next = NULL;
- }
-
- /* Now that we've walked the whole filesystem list, anything left
- * over in the database lists has been deleted. */
- while (notmuch_filenames_valid (db_files))
- {
- char *absolute = talloc_asprintf (state->removed_files,
- "%s/%s", path,
- notmuch_filenames_get (db_files));
-
- _filename_list_add (state->removed_files, absolute);
-
- notmuch_filenames_move_to_next (db_files);
- }
-
- while (notmuch_filenames_valid (db_subdirs))
- {
- char *absolute = talloc_asprintf (state->removed_directories,
- "%s/%s", path,
- notmuch_filenames_get (db_subdirs));
-
- _filename_list_add (state->removed_directories, absolute);
-
- notmuch_filenames_move_to_next (db_subdirs);
- }
-
- if (! interrupted) {
- status = notmuch_directory_set_mtime (directory, fs_mtime);
- if (status && ret == NOTMUCH_STATUS_SUCCESS)
- ret = status;
- }
-
- DONE:
- if (next)
- talloc_free (next);
- if (entry)
- free (entry);
- if (dir)
- closedir (dir);
- if (fs_entries)
- free (fs_entries);
- if (db_subdirs)
- notmuch_filenames_destroy (db_subdirs);
- if (db_files)
- notmuch_filenames_destroy (db_files);
- if (directory)
- notmuch_directory_destroy (directory);
-
- return ret;
-}
-
/* This is the top-level entry point for add_files. It does a couple
* of error checks, sets up the progress-printing timer and then calls
* into the recursive function. */
static notmuch_status_t
-add_files (notmuch_database_t *notmuch,
+add_files (notmuch_mailstore_t *mailstore,
const char *path,
- add_files_state_t *state)
+ notmuch_indexing_context_t *state) /* FIXME: rename */
{
notmuch_status_t status;
struct sigaction action;
@@ -517,6 +84,9 @@ add_files (notmuch_database_t *notmuch,
notmuch_bool_t timer_is_active = FALSE;
struct stat st;
+ state->print_progress = 0;
+ state->print_progress_cb = add_files_print_progress;
+
if (state->output_is_a_tty && ! debugger_is_active () && ! state->verbose) {
/* Setup our handler for SIGALRM */
memset (&action, 0, sizeof (struct sigaction));
@@ -546,7 +116,7 @@ add_files (notmuch_database_t *notmuch,
return NOTMUCH_STATUS_FILE_ERROR;
}
- status = add_files_recursive (notmuch, path, state);
+ status = notmuch_mailstore_index_new (mailstore, path, state);
if (timer_is_active) {
/* Now stop the timer. */
@@ -564,80 +134,11 @@ add_files (notmuch_database_t *notmuch,
return status;
}
-/* XXX: This should be merged with the add_files function since it
- * shares a lot of logic with it. */
-/* Recursively count all regular files in path and all sub-directories
- * of path. The result is added to *count (which should be
- * initialized to zero by the top-level caller before calling
- * count_files). */
-static void
-count_files (const char *path, int *count)
-{
- struct dirent *entry = NULL;
- char *next;
- struct stat st;
- struct dirent **fs_entries = NULL;
- int num_fs_entries = scandir (path, &fs_entries, 0, dirent_sort_inode);
- int i = 0;
-
- if (num_fs_entries == -1) {
- fprintf (stderr, "Warning: failed to open directory %s: %s\n",
- path, strerror (errno));
- goto DONE;
- }
-
- while (!interrupted) {
- if (i == num_fs_entries)
- break;
-
- entry = fs_entries[i++];
-
- /* Ignore special directories to avoid infinite recursion.
- * Also ignore the .notmuch directory.
- */
- /* XXX: Eventually we'll want more sophistication to let the
- * user specify files to be ignored. */
- if (strcmp (entry->d_name, ".") == 0 ||
- strcmp (entry->d_name, "..") == 0 ||
- strcmp (entry->d_name, ".notmuch") == 0)
- {
- continue;
- }
-
- if (asprintf (&next, "%s/%s", path, entry->d_name) == -1) {
- next = NULL;
- fprintf (stderr, "Error descending from %s to %s: Out of memory\n",
- path, entry->d_name);
- continue;
- }
-
- stat (next, &st);
-
- if (S_ISREG (st.st_mode)) {
- *count = *count + 1;
- if (*count % 1000 == 0) {
- printf ("Found %d files so far.\r", *count);
- fflush (stdout);
- }
- } else if (S_ISDIR (st.st_mode)) {
- count_files (next, count);
- }
-
- free (next);
- }
-
- DONE:
- if (entry)
- free (entry);
- if (fs_entries)
- free (fs_entries);
-}
-
static void
upgrade_print_progress (void *closure,
double progress)
{
- add_files_state_t *state = closure;
+ notmuch_indexing_context_t *state = closure;
printf ("Upgrading database: %.2f%% complete", progress * 100.0);
@@ -659,55 +160,12 @@ upgrade_print_progress (void *closure,
fflush (stdout);
}
-/* Recursively remove all filenames from the database referring to
- * 'path' (or to any of its children). */
-static void
-_remove_directory (void *ctx,
- notmuch_database_t *notmuch,
- const char *path,
- int *renamed_files,
- int *removed_files)
-{
- notmuch_directory_t *directory;
- notmuch_filenames_t *files, *subdirs;
- notmuch_status_t status;
- char *absolute;
-
- directory = notmuch_database_get_directory (notmuch, path);
-
- for (files = notmuch_directory_get_child_files (directory);
- notmuch_filenames_valid (files);
- notmuch_filenames_move_to_next (files))
- {
- absolute = talloc_asprintf (ctx, "%s/%s", path,
- notmuch_filenames_get (files));
- status = notmuch_database_remove_message (notmuch, absolute);
- if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID)
- *renamed_files = *renamed_files + 1;
- else
- *removed_files = *removed_files + 1;
- talloc_free (absolute);
- }
-
- for (subdirs = notmuch_directory_get_child_directories (directory);
- notmuch_filenames_valid (subdirs);
- notmuch_filenames_move_to_next (subdirs))
- {
- absolute = talloc_asprintf (ctx, "%s/%s", path,
- notmuch_filenames_get (subdirs));
- _remove_directory (ctx, notmuch, absolute, renamed_files, removed_files);
- talloc_free (absolute);
- }
-
- notmuch_directory_destroy (directory);
-}
-
int
notmuch_new_command (void *ctx, int argc, char *argv[])
{
notmuch_config_t *config;
notmuch_database_t *notmuch;
- add_files_state_t add_files_state;
+ notmuch_indexing_context_t add_files_state; /* FIXME: Rename */
double elapsed;
struct timeval tv_now;
int ret = 0;
@@ -715,10 +173,10 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
const char *db_path;
char *dot_notmuch_path;
struct sigaction action;
- _filename_node_t *f;
- int renamed_files, removed_files;
- notmuch_status_t status;
int i;
+ notmuch_mailstore_t *mailstore;
+
+ indexing_ctx = &add_files_state;
add_files_state.verbose = 0;
add_files_state.output_is_a_tty = isatty (fileno (stdout));
@@ -737,6 +195,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
return 1;
db_path = notmuch_config_get_database_path (config);
+ mailstore = notmuch_config_get_mailstore (config);
dot_notmuch_path = talloc_asprintf (ctx, "%s/%s", db_path, ".notmuch");
@@ -744,7 +203,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
int count;
count = 0;
- count_files (db_path, &count);
+ notmuch_mailstore_count_files (mailstore, db_path, &count, &interrupted);
if (interrupted)
return 1;
@@ -777,6 +236,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
/* Setup our handler for SIGINT. We do this after having
* potentially done a database upgrade we this interrupt handler
* won't support. */
+ add_files_state.interrupted = 0;
memset (&action, 0, sizeof (struct sigaction));
action.sa_handler = handle_sigint;
sigemptyset (&action.sa_mask);
@@ -790,28 +250,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
add_files_state.added_messages = 0;
gettimeofday (&add_files_state.tv_start, NULL);
- add_files_state.removed_files = _filename_list_create (ctx);
- add_files_state.removed_directories = _filename_list_create (ctx);
-
- ret = add_files (notmuch, db_path, &add_files_state);
-
- removed_files = 0;
- renamed_files = 0;
- for (f = add_files_state.removed_files->head; f; f = f->next) {
- status = notmuch_database_remove_message (notmuch, f->filename);
- if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID)
- renamed_files++;
- else
- removed_files++;
- }
-
- for (f = add_files_state.removed_directories->head; f; f = f->next) {
- _remove_directory (ctx, notmuch, f->filename,
- &renamed_files, &removed_files);
- }
-
- talloc_free (add_files_state.removed_files);
- talloc_free (add_files_state.removed_directories);
+ ret = add_files (mailstore, db_path, &add_files_state);
gettimeofday (&tv_now, NULL);
elapsed = notmuch_time_elapsed (add_files_state.tv_start,
@@ -839,16 +278,16 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
printf ("No new mail.");
}
- if (removed_files) {
+ if (add_files_state.removed_files) {
printf (" Removed %d %s.",
- removed_files,
- removed_files == 1 ? "message" : "messages");
+ add_files_state.removed_files,
+ add_files_state.removed_files == 1 ? "message" : "messages");
}
- if (renamed_files) {
+ if (add_files_state.renamed_files) {
printf (" Detected %d file %s.",
- renamed_files,
- renamed_files == 1 ? "rename" : "renames");
+ add_files_state.renamed_files,
+ add_files_state.renamed_files == 1 ? "rename" : "renames");
}
printf ("\n");
--
1.7.0
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [PATCH 3/4] Add maildir-based mailstore
2010-03-18 15:39 Mailstore abstraction & maildir synchronization Michal Sojka
2010-03-18 15:39 ` [PATCH 1/4] Mailstore abstraction interface Michal Sojka
2010-03-18 15:39 ` [PATCH 2/4] Convert mailstore abstraction Michal Sojka
@ 2010-03-18 15:39 ` Michal Sojka
2010-03-23 10:56 ` Ruben Pollan
2010-03-18 15:39 ` [PATCH " Michal Sojka
2010-03-24 5:06 ` Mailstore abstraction & maildir synchronization Stewart Smith
4 siblings, 1 reply; 25+ messages in thread
From: Michal Sojka @ 2010-03-18 15:39 UTC (permalink / raw)
To: notmuch
This mailstore allows bi-directional synchronization between maildir
flags and certain tags. The flag-to-tag mapping is defined by flag2tag
array.
The synchronization works this way:
1) Whenever notmuch new is executed, then for every new/renamed
message the tags defined in flag2tag are either added or removed
depending on the flags from the file name.
2) Whenever notmuch tag is executed, a new set of flags based on the
tags is constructed and if the new flags differ from that stored in
the file name, the file is renamed and notmuch database is updated
to contain the new name for the file.
This mailstore is enabled by putting
[mailstore]
type=maildir
to your .notmuch-config.
Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz>
---
lib/database.cc | 7 ++
lib/mailstore-files.c | 167 ++++++++++++++++++++++++++++++++++++++++++++++++-
lib/mailstore.c | 1 +
lib/message.cc | 41 ++++++++++++-
lib/notmuch-private.h | 4 +
lib/notmuch.h | 1 +
6 files changed, 217 insertions(+), 4 deletions(-)
diff --git a/lib/database.cc b/lib/database.cc
index 93c8d0f..33ef889 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1471,6 +1471,13 @@ notmuch_database_add_message (notmuch_database_t *notmuch,
_notmuch_message_add_filename (message, filename);
+ /* This is a new message or it has a new filename and as such,
+ * its tags in database either do not exists or might be out
+ * of date. Mailstore assigns the tags later in index_new(),
+ * but until then we should not synchronize the tags back to
+ * the mailstore. */
+ notmuch_message_set_flag(message, NOTMUCH_MESSAGE_FLAG_TAGS_INVALID, TRUE);
+
/* Is this a newly created message object? */
if (private_status == NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND) {
_notmuch_message_add_term (message, "type", "mail");
diff --git a/lib/mailstore-files.c b/lib/mailstore-files.c
index ace2664..d89e183 100644
--- a/lib/mailstore-files.c
+++ b/lib/mailstore-files.c
@@ -24,6 +24,9 @@
#include "notmuch.h"
#include "mailstore-private.h"
#include <dirent.h>
+#include <stdbool.h>
+
+#define ARRAY_SIZE(arr) (sizeof (arr) / sizeof (arr[0]))
typedef struct _filename_node {
char *filename;
@@ -69,8 +72,9 @@ _filename_list_add (_filename_list_t *list,
}
static void
-tag_inbox_and_unread (notmuch_message_t *message)
+tag_inbox_and_unread (notmuch_message_t *message, const char *filename)
{
+ (void)filename;
notmuch_message_add_tag (message, "inbox");
notmuch_message_add_tag (message, "unread");
}
@@ -117,6 +121,141 @@ _entries_resemble_maildir (struct dirent **entries, int count)
}
+struct mailstore_priv {
+ void (*tag_new)(notmuch_message_t *message, const char *filename);
+ void (*tag_renamed)(notmuch_message_t *message, const char *filename);
+};
+
+struct maildir_flag_tag {
+ char flag;
+ const char *tag;
+ bool inverse;
+};
+
+/* ASCII ordered table of Maildir flags and assiciated tags */
+struct maildir_flag_tag flag2tag[] = {
+ { 'D', "draft", false},
+ { 'F', "flagged", false},
+ { 'P', "passed", false},
+ { 'R', "replied", false},
+ { 'S', "unread", true },
+ { 'T', "delete", false},
+};
+
+static void
+tag_from_maildir_flags(notmuch_message_t *message, const char *filename)
+{
+ const char *flags, *p;
+ char f;
+ bool valid, unread;
+ unsigned i;
+
+ flags = strstr(filename, ":2,");
+ if (!flags)
+ return;
+ flags += 3;
+
+ /* Check that the letters are valid Maildir flags */
+ f = 0;
+ valid = true;
+ for (p=flags; valid && *p; p++) {
+ switch (*p) {
+ case 'P':
+ case 'R':
+ case 'S':
+ case 'T':
+ case 'D':
+ case 'F':
+ if (*p > f) f=*p;
+ else valid = false;
+ break;
+ default:
+ valid = false;
+ }
+ }
+ if (!valid) {
+ /* fprintf(stderr, "Note: Invalid maildir flags: %s\n", message->filename); */
+ return;
+ }
+
+ notmuch_message_freeze(message);
+ unread = true;
+ for (i = 0; i < ARRAY_SIZE(flag2tag); i++) {
+ if ((strchr(flags, flag2tag[i].flag) != NULL) ^ flag2tag[i].inverse) {
+ notmuch_message_add_tag (message, flag2tag[i].tag);
+ } else {
+ notmuch_message_remove_tag (message, flag2tag[i].tag);
+ }
+ }
+ notmuch_message_thaw(message);
+
+ /* From now on, we can synchronize the tags from the database to
+ * the mailstore. */
+ notmuch_message_set_flag(message, NOTMUCH_MESSAGE_FLAG_TAGS_INVALID, FALSE);
+}
+
+/* Store maildir-related tags as maildir flags */
+static notmuch_private_status_t
+maildir_sync_tags(notmuch_mailstore_t *mailstore,
+ notmuch_message_t *message)
+{
+ notmuch_tags_t *tags;
+ const char *tag;
+ char flags[ARRAY_SIZE(flag2tag)+1];
+ unsigned i;
+ char *p;
+ const char *filename;
+ char *filename_new;
+
+ (void)mailstore;
+ for (i = 0; i < ARRAY_SIZE(flag2tag); i++)
+ flags[i] = flag2tag[i].inverse ? flag2tag[i].flag : '\0';
+
+ for (tags = notmuch_message_get_tags (message);
+ notmuch_tags_valid (tags);
+ notmuch_tags_move_to_next (tags))
+ {
+ tag = notmuch_tags_get (tags);
+ for (i = 0; i < ARRAY_SIZE(flag2tag); i++) {
+ if (strcmp(tag, flag2tag[i].tag) == 0)
+ flags[i] = flag2tag[i].inverse ? '\0' : flag2tag[i].flag;
+ }
+ }
+
+ p = flags;
+ for (i = 0; i < ARRAY_SIZE(flag2tag); i++) {
+ if (flags[i])
+ *p++ = flags[i];
+ }
+ *p = '\0';
+
+ filename = notmuch_message_get_filename(message);
+ p = strstr(filename, ":2,");
+ if (!p)
+ return NOTMUCH_STATUS_SUCCESS; /* FIXME: Is this OK? */
+ p += 3;
+ filename_new = talloc_size(message, (p-filename) + sizeof(flags));
+ if (unlikely (filename_new == NULL))
+ return NOTMUCH_STATUS_OUT_OF_MEMORY;
+ memcpy(filename_new, filename, p-filename);
+ strcpy(filename_new+(p-filename), flags);
+
+ if (strcmp (filename, filename_new) != 0) {
+ rename (filename, filename_new);
+
+ return _notmuch_message_rename (message, filename_new);
+ /* _notmuch_message_sync is our caller. Do not call it here. */
+ }
+ return NOTMUCH_STATUS_SUCCESS;
+}
+
+static void
+tag_inbox_and_maildir_flags (notmuch_message_t *message, const char *filename)
+{
+ notmuch_message_add_tag (message, "inbox");
+ tag_from_maildir_flags(message, filename);
+}
+
/* Examine 'path' recursively as follows:
*
* o Ask the filesystem for the mtime of 'path' (fs_mtime)
@@ -171,6 +310,7 @@ add_files_recursive (notmuch_mailstore_t *mailstore,
struct stat st;
notmuch_bool_t is_maildir, new_directory;
_indexing_context_priv_t *priv = state->priv;
+ struct mailstore_priv *mailstore_priv = mailstore->priv;
notmuch_database_t *notmuch = mailstore->notmuch;
if (stat (path, &st)) {
@@ -362,11 +502,12 @@ add_files_recursive (notmuch_mailstore_t *mailstore,
/* success */
case NOTMUCH_STATUS_SUCCESS:
state->added_messages++;
- tag_inbox_and_unread (message);
+ mailstore_priv->tag_new (message, next);
break;
/* Non-fatal issues (go on to next file) */
case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
- /* Stay silent on this one. */
+ if (mailstore_priv->tag_renamed)
+ mailstore_priv->tag_renamed (message, next);
break;
case NOTMUCH_STATUS_FILE_NOT_EMAIL:
fprintf (stderr, "Note: Ignoring non-mail file: %s\n",
@@ -623,6 +764,10 @@ open_file(notmuch_mailstore_t *mailstore, const char *filename)
return fopen (abs_filename, "r");
}
+struct mailstore_priv files_priv = {
+ .tag_new = tag_inbox_and_unread,
+};
+
/* Original notmuch mail store */
struct _notmuch_mailstore mailstore_files = {
.type = "files",
@@ -630,4 +775,20 @@ struct _notmuch_mailstore mailstore_files = {
.index_new = index_new,
.sync_tags = NULL,
.open_file = open_file,
+ .priv = &files_priv,
+};
+
+struct mailstore_priv maildir_priv = {
+ .tag_new = tag_inbox_and_maildir_flags,
+ .tag_renamed = tag_from_maildir_flags,
+};
+
+/* Similar to mailstore_files, but does bi-directional synchronization between certain tags and maildir flags */
+struct _notmuch_mailstore mailstore_maildir = {
+ .type = "maildir",
+ .count_files = count_files,
+ .index_new = index_new,
+ .sync_tags = maildir_sync_tags,
+ .open_file = open_file,
+ .priv = &maildir_priv,
};
diff --git a/lib/mailstore.c b/lib/mailstore.c
index cdac9a6..7dc55ad 100644
--- a/lib/mailstore.c
+++ b/lib/mailstore.c
@@ -25,6 +25,7 @@
static notmuch_mailstore_t *available[] = {
&mailstore_files,
+ &mailstore_maildir,
};
notmuch_mailstore_t *
diff --git a/lib/message.cc b/lib/message.cc
index c32ee7d..51208ba 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -561,7 +561,8 @@ _notmuch_message_sync (notmuch_message_t *message)
if (message->notmuch->mode == NOTMUCH_DATABASE_MODE_READ_ONLY)
return;
- if (message->notmuch->mailstore->sync_tags) {
+ if (message->notmuch->mailstore->sync_tags &&
+ !notmuch_message_get_flag(message, NOTMUCH_MESSAGE_FLAG_TAGS_INVALID)) {
status = message->notmuch->mailstore->sync_tags (message->notmuch->mailstore,
message);
if (status != NOTMUCH_PRIVATE_STATUS_SUCCESS) {
@@ -683,6 +684,44 @@ _notmuch_message_remove_term (notmuch_message_t *message,
return NOTMUCH_PRIVATE_STATUS_SUCCESS;
}
+/* Change the message filename stored in the database.
+ *
+ * This change will not be reflected in the database until the next
+ * call to _notmuch_message_sync.
+ */
+notmuch_private_status_t
+_notmuch_message_rename (notmuch_message_t *message,
+ const char *new_filename)
+{
+ void *local = talloc_new (message);
+ char *direntry;
+ Xapian::PostingIterator i, end;
+ Xapian::Document document;
+ notmuch_private_status_t pstatus;
+ notmuch_status_t status;
+ const char *old_filename;
+
+ old_filename = notmuch_message_get_filename(message);
+ old_filename = talloc_reference(local, old_filename);
+ if (unlikely(!old_filename))
+ return NOTMUCH_PRIVATE_STATUS_OUT_OF_MEMORY;
+
+ status = _notmuch_message_add_filename (message, new_filename);
+ if (status)
+ return (notmuch_private_status_t)status;
+
+ status = _notmuch_database_filename_to_direntry (local, message->notmuch,
+ old_filename, &direntry);
+ if (status)
+ return (notmuch_private_status_t)status;
+
+ pstatus = _notmuch_message_remove_term (message, "file-direntry", direntry);
+
+ talloc_free (local);
+
+ return pstatus;
+}
+
notmuch_status_t
notmuch_message_add_tag (notmuch_message_t *message, const char *tag)
{
diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index d52d84d..63e75ec 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -245,6 +245,10 @@ notmuch_status_t
_notmuch_message_add_filename (notmuch_message_t *message,
const char *filename);
+notmuch_private_status_t
+_notmuch_message_rename (notmuch_message_t *message,
+ const char *new_filename);
+
void
_notmuch_message_ensure_thread_id (notmuch_message_t *message);
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 4f4cc8b..eab909b 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -761,6 +761,7 @@ notmuch_message_get_filename (notmuch_message_t *message);
/* Message flags */
typedef enum _notmuch_message_flag {
NOTMUCH_MESSAGE_FLAG_MATCH,
+ NOTMUCH_MESSAGE_FLAG_TAGS_INVALID,
} notmuch_message_flag_t;
/* Get a value of a flag for the email corresponding to 'message'. */
--
1.7.0
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [PATCH 4/4] Tests for maildir-based mailstore
2010-03-18 15:39 Mailstore abstraction & maildir synchronization Michal Sojka
` (2 preceding siblings ...)
2010-03-18 15:39 ` [PATCH 3/4] Add maildir-based mailstore Michal Sojka
@ 2010-03-18 15:39 ` Michal Sojka
2010-03-23 10:59 ` Ruben Pollan
2010-03-24 5:06 ` Mailstore abstraction & maildir synchronization Stewart Smith
4 siblings, 1 reply; 25+ messages in thread
From: Michal Sojka @ 2010-03-18 15:39 UTC (permalink / raw)
To: notmuch
Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz>
---
test/t0006-maildir.sh | 113 +++++++++++++++++++++++++++++++++++++++++++++++++
test/test-lib.sh | 7 ++-
2 files changed, 118 insertions(+), 2 deletions(-)
create mode 100755 test/t0006-maildir.sh
diff --git a/test/t0006-maildir.sh b/test/t0006-maildir.sh
new file mode 100755
index 0000000..e584908
--- /dev/null
+++ b/test/t0006-maildir.sh
@@ -0,0 +1,113 @@
+#!/bin/bash
+
+test_description="Test maildir mailstore"
+
+. ./test-lib.sh
+
+filter_output() {
+ grep -v -E -e "$NOTMUCH_IGNORED_OUTPUT_REGEXP" | sed -e "$NOTMUCH_THREAD_ID_SQUELCH"
+}
+
+filter_show() {
+ sed -e 's/, /,\n/g'|sed -e '/^"filename"/ s/:2,[A-Z]*//' -e '/^"tags"/d'
+}
+
+cat >> "$NOTMUCH_CONFIG" <<EOF
+[mailstore]
+type=maildir
+EOF
+
+test_expect_success "No new mail" '
+execute_expecting new "No new mail."
+'
+cat > expected <<EOF
+Added 1 new message to the database.
+EOF
+test_expect_success "Add a message, no flags" '
+generate_message [subject]="\"test message\"" [date]="\"Sat, 01 Jan 2000 12:00:00 -0000\"" [filename]="\"msg-001:2,\"" &&
+notmuch new | filter_output > actual &&
+test_cmp expected actual
+#emacs --eval "(gdb \"gdb --annotate=3 --args $(which notmuch) new\")"
+'
+cat > expected <<EOF
+thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test message (inbox unread)
+EOF
+test_expect_success 'Search for the message' '
+notmuch search tag:inbox and tag:unread | filter_output > actual &&
+test_cmp expected actual
+'
+cat > expected <<EOF
+No new mail. Detected 1 file rename.
+EOF
+test_expect_success 'Add seen flag' '
+mv "${gen_msg_filename}" "${gen_msg_filename}S" &&
+increment_mtime "$(dirname "${gen_msg_filename}")" &&
+notmuch new | filter_output > actual &&
+test_cmp expected actual
+'
+cat > expected <<EOF
+thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test message (inbox)
+EOF
+test_expect_success 'Check that tags were updated' '
+notmuch search tag:inbox and not tag:unread | filter_output > actual &&
+test_cmp expected actual
+'
+cat > expected <<EOF
+Added 1 new message to the database.
+EOF
+test_expect_success "Add a seen message" '
+generate_message [subject]="\"test message2\"" [date]="\"Sat, 01 Jan 2000 12:00:00 -0000\"" [filename]="\"msg-002:2,S\"" &&
+notmuch new | filter_output > actual &&
+test_cmp expected actual
+'
+cat > expected <<EOF
+thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test message (inbox)
+thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test message2 (inbox)
+EOF
+test_expect_success 'Check that the seen message is not tagged unread' '
+notmuch search tag:inbox and not tag:unread | filter_output > actual &&
+test_cmp expected actual
+'
+test_expect_success 'Tag the seen messages as replied' '
+notmuch tag +replied -inbox tag:inbox and not tag:unread
+'
+
+cat > expected <<EOF
+msg-001:2,RS
+msg-002:2,RS
+EOF
+test_expect_success 'Check that R flag was added (rename)' '
+ls -1 "${MAIL_DIR}" > actual &&
+test_cmp expected actual
+'
+echo -n '[[[{"id": "msg-001@notmuch-test-suite",
+"match": true,
+"filename": "/home/wsh/src/notmuch/test/trash directory.t0006-maildir/mail/msg-001",
+"headers": {"Subject": "test message",
+"From": "Notmuch Test Suite <test_suite@notmuchmail.org>",
+"To": "Notmuch Test Suite <test_suite@notmuchmail.org>",
+"Cc": "",
+"Bcc": "",
+"Date": "Sat,
+01 Jan 2000 12:00:00 -0000"},
+"body": [{"id": 1,
+"content-type": "text/plain",
+"content": "This is just a test message at /home/wsh/src/notmuch/test/trash directory.t0006-maildir/mail/msg-001:2,\n"}]},
+[]]]]' > show-expected
+
+test_expect_success 'Renamed message can be shown without running notmuch new' '
+notmuch show --format=json id:msg-001@notmuch-test-suite | filter_show > show-actual &&
+test_cmp show-expected show-actual
+'
+
+test_expect_success 'Test that we can reply to the renamed message' '
+notmuch reply id:msg-001@notmuch-test-suite
+'
+
+echo "No new mail." > expected
+test_expect_success 'No rename should be detected by notmuch new' '
+increment_mtime "$(dirname "${gen_msg_filename}")" &&
+notmuch new > actual &&
+test_cmp expected actual
+'
+test_done
diff --git a/test/test-lib.sh b/test/test-lib.sh
index 5417fe7..917631b 100755
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -257,8 +257,11 @@ generate_message ()
local additional_headers
gen_msg_cnt=$((gen_msg_cnt + 1))
- gen_msg_name=msg-$(printf "%03d" $gen_msg_cnt)
- gen_msg_id="${gen_msg_name}@notmuch-test-suite"
+ if [ -z "${template[filename]}" ]; then
+ template[filename]="msg-$(printf "%03d" $gen_msg_cnt)"
+ fi
+ gen_msg_name=${template[filename]}
+ gen_msg_id="${gen_msg_name%:2,*}@notmuch-test-suite"
if [ -z "${template[dir]}" ]; then
gen_msg_filename="${MAIL_DIR}/$gen_msg_name"
--
1.7.0
^ permalink raw reply related [flat|nested] 25+ messages in thread
* Re: [PATCH 3/4] Add maildir-based mailstore
2010-03-18 15:39 ` [PATCH 3/4] Add maildir-based mailstore Michal Sojka
@ 2010-03-23 10:56 ` Ruben Pollan
2010-03-23 13:35 ` Michal Sojka
0 siblings, 1 reply; 25+ messages in thread
From: Ruben Pollan @ 2010-03-23 10:56 UTC (permalink / raw)
To: Michal Sojka; +Cc: notmuch
[-- Attachment #1: Type: text/plain, Size: 757 bytes --]
On 16:39, Thu 18 Mar 10, Michal Sojka wrote:
> This mailstore allows bi-directional synchronization between maildir
> flags and certain tags. The flag-to-tag mapping is defined by flag2tag
> array.
I'm trying it, the synchronization from my maildir to notmuch seems to work fine
except for the "unread" tag, it is never set. I use normally my maildir with
mutt, so the unread messages are on mail_folder/new. Could that be the problem?
is it a bug or something I do wrong?
Thanks for your patch.
--
Rubén Pollán | jabber:meskio@jabber.org
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Hay un mundo
a la vuelta de la esquina de tu mente,
donde la realidad es un intruso
y los sueños se hacen realidad.
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH 4/4] Tests for maildir-based mailstore
2010-03-18 15:39 ` [PATCH " Michal Sojka
@ 2010-03-23 10:59 ` Ruben Pollan
2010-03-23 13:47 ` Michal Sojka
0 siblings, 1 reply; 25+ messages in thread
From: Ruben Pollan @ 2010-03-23 10:59 UTC (permalink / raw)
To: Michal Sojka; +Cc: notmuch
[-- Attachment #1: Type: text/plain, Size: 618 bytes --]
On 16:39, Thu 18 Mar 10, Michal Sojka wrote:
> Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz>
> ---
> test/t0006-maildir.sh | 113 +++++++++++++++++++++++++++++++++++++++++++++++++
> test/test-lib.sh | 7 ++-
> 2 files changed, 118 insertions(+), 2 deletions(-)
> create mode 100755 test/t0006-maildir.sh
This patch is not applicable on HEAD. The file test/test-lib.sh don't exists.
--
Rubén Pollán | jabber:meskio@jabber.org
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
No vamos a reivindicar nada,
no vamos a pedir nada.
Tomaremos, okuparemos.
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH 3/4] Add maildir-based mailstore
2010-03-23 10:56 ` Ruben Pollan
@ 2010-03-23 13:35 ` Michal Sojka
2010-03-23 14:37 ` Ruben Pollan
0 siblings, 1 reply; 25+ messages in thread
From: Michal Sojka @ 2010-03-23 13:35 UTC (permalink / raw)
To: Ruben Pollan; +Cc: notmuch
On Tue, 23 Mar 2010, Ruben Pollan wrote:
> On 16:39, Thu 18 Mar 10, Michal Sojka wrote:
> > This mailstore allows bi-directional synchronization between maildir
> > flags and certain tags. The flag-to-tag mapping is defined by flag2tag
> > array.
>
> I'm trying it, the synchronization from my maildir to notmuch seems to work fine
> except for the "unread" tag, it is never set. I use normally my maildir with
> mutt, so the unread messages are on mail_folder/new. Could that be the problem?
> is it a bug or something I do wrong?
Yes, that's a bug. Messages in your mail_folder/new probably don't have
any 'info' (flags) in their filename, which is correct with respect to
http://cr.yp.to/proto/maildir.html. My patch incorrectly ignores the
files without 'info'. I didn't catch this error because is seems that
offlineimap puts 'info' part even to messages in new/.
I think that the correct fix for this is to change how notmuch new
works: Whenever a message in new/ is found, it will be moved to cur/ and
in there is no info part it will be appended and the message will be
tagged accordingly.
I'll try to implement it and send a new patch.
-Michal
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH 4/4] Tests for maildir-based mailstore
2010-03-23 10:59 ` Ruben Pollan
@ 2010-03-23 13:47 ` Michal Sojka
0 siblings, 0 replies; 25+ messages in thread
From: Michal Sojka @ 2010-03-23 13:47 UTC (permalink / raw)
To: Ruben Pollan; +Cc: notmuch
On Tue, 23 Mar 2010, Ruben Pollan wrote:
> On 16:39, Thu 18 Mar 10, Michal Sojka wrote:
> > Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz>
> > ---
> > test/t0006-maildir.sh | 113 +++++++++++++++++++++++++++++++++++++++++++++++++
> > test/test-lib.sh | 7 ++-
> > 2 files changed, 118 insertions(+), 2 deletions(-)
> > create mode 100755 test/t0006-maildir.sh
>
> This patch is not applicable on HEAD. The file test/test-lib.sh don't exists.
Yes, you need to apply patches also patches for the test framework. The
whole patch series can be seen at
http://rtime.felk.cvut.cz/gitweb/notmuch.git/shortlog/refs/heads/mailstore-abstraction-v1
and can be merged by
git pull git://rtime.felk.cvut.cz/notmuch.git mailstore-abstraction-v1
I have to (try to) finish relicencing of the git's test framework from
GPLv2 to GPLv>=2, so that it can be merged to notmuch.
-M
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH 3/4] Add maildir-based mailstore
2010-03-23 13:35 ` Michal Sojka
@ 2010-03-23 14:37 ` Ruben Pollan
2010-03-26 21:41 ` Michal Sojka
0 siblings, 1 reply; 25+ messages in thread
From: Ruben Pollan @ 2010-03-23 14:37 UTC (permalink / raw)
To: Michal Sojka; +Cc: notmuch
[-- Attachment #1: Type: text/plain, Size: 1795 bytes --]
On 14:35, Tue 23 Mar 10, Michal Sojka wrote:
> On Tue, 23 Mar 2010, Ruben Pollan wrote:
> > On 16:39, Thu 18 Mar 10, Michal Sojka wrote:
> > > This mailstore allows bi-directional synchronization between maildir
> > > flags and certain tags. The flag-to-tag mapping is defined by flag2tag
> > > array.
> >
> > I'm trying it, the synchronization from my maildir to notmuch seems to work fine
> > except for the "unread" tag, it is never set. I use normally my maildir with
> > mutt, so the unread messages are on mail_folder/new. Could that be the problem?
> > is it a bug or something I do wrong?
>
> Yes, that's a bug. Messages in your mail_folder/new probably don't have
> any 'info' (flags) in their filename, which is correct with respect to
> http://cr.yp.to/proto/maildir.html. My patch incorrectly ignores the
> files without 'info'. I didn't catch this error because is seems that
> offlineimap puts 'info' part even to messages in new/.
True, I use fetchmail for download my email, it put the email in the new folder
without 'info'.
> I think that the correct fix for this is to change how notmuch new
> works: Whenever a message in new/ is found, it will be moved to cur/ and
> in there is no info part it will be appended and the message will be
> tagged accordingly.
I think is better if new keeps the email where it is, only updates the database,
and when you remove the unread tag move the message from folder to folder. Some
mail readers (at least mutt) handles different the emails on the new folder than
the ones with no "S" flag con cur folder.
> I'll try to implement it and send a new patch.
That will be nice.
--
Rubén Pollán | jabber:meskio@jabber.org
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Nos vamos a Croatan.
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Mailstore abstraction & maildir synchronization
2010-03-18 15:39 Mailstore abstraction & maildir synchronization Michal Sojka
` (3 preceding siblings ...)
2010-03-18 15:39 ` [PATCH " Michal Sojka
@ 2010-03-24 5:06 ` Stewart Smith
2010-03-26 22:08 ` Michal Sojka
4 siblings, 1 reply; 25+ messages in thread
From: Stewart Smith @ 2010-03-24 5:06 UTC (permalink / raw)
To: Michal Sojka, notmuch
On Thu, 18 Mar 2010 16:39:36 +0100, Michal Sojka <sojkam1@fel.cvut.cz> wrote:
> - Only file-based storage is suported. Notmuch access the files
> directly, and not via the mailstore interface.
It'll be great when this is fixed... should be trivial to add a git
backend then.
(i have in no way been looking at tags in git though... doesn't really
interest me and git aint a database)
> - (maildir) Viewing/storing of attachments of unread messages doesn't
> work. The reason is that when you view the message it its unread tag
> is removed which leads to rename of the file, but Emacs still uses
> the original name to access the attachment.
What about migrating from a maildir that's turned into notmuch back to
this maildir backend? What will be authoritive: maildir or notmuch database?
--
Stewart Smith
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: [PATCH 3/4] Add maildir-based mailstore
2010-03-23 14:37 ` Ruben Pollan
@ 2010-03-26 21:41 ` Michal Sojka
2010-03-26 21:42 ` [PATCH v2 1/4] Mailstore abstraction interface Michal Sojka
` (3 more replies)
0 siblings, 4 replies; 25+ messages in thread
From: Michal Sojka @ 2010-03-26 21:41 UTC (permalink / raw)
To: Ruben Pollan; +Cc: notmuch
On Tue, 23 Mar 2010, Ruben Pollan wrote:
> On 14:35, Tue 23 Mar 10, Michal Sojka wrote:
> > Yes, that's a bug. Messages in your mail_folder/new probably don't have
> > any 'info' (flags) in their filename, which is correct with respect to
> > http://cr.yp.to/proto/maildir.html. My patch incorrectly ignores the
> > files without 'info'. I didn't catch this error because is seems that
> > offlineimap puts 'info' part even to messages in new/.
>
> True, I use fetchmail for download my email, it put the email in the new folder
> without 'info'.
>
> > I think that the correct fix for this is to change how notmuch new
> > works: Whenever a message in new/ is found, it will be moved to cur/ and
> > in there is no info part it will be appended and the message will be
> > tagged accordingly.
>
> I think is better if new keeps the email where it is, only updates the database,
> and when you remove the unread tag move the message from folder to folder. Some
> mail readers (at least mutt) handles different the emails on the new folder than
> the ones with no "S" flag con cur folder.
>
> > I'll try to implement it and send a new patch.
>
> That will be nice.
Hi,
Here is the updated patch series implementing your proposal. Let me know
if it solves your problem.
I also did some minor cleanups and included "Prevent data loss caused by
SIGINT during notmuch new" patch which I've just sent separately.
-Michal
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v2 1/4] Mailstore abstraction interface
2010-03-26 21:41 ` Michal Sojka
@ 2010-03-26 21:42 ` Michal Sojka
2010-03-26 21:42 ` [PATCH v2 2/4] Conversion to mailstore abstraction Michal Sojka
` (2 subsequent siblings)
3 siblings, 0 replies; 25+ messages in thread
From: Michal Sojka @ 2010-03-26 21:42 UTC (permalink / raw)
To: notmuch
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 23517 bytes --]
The goal of mailstore abstraction is to allow notmuch to store tags
together with email messages. The abstract interface is needed because
people want to use different ways of storing their emails. This
patchseries implements two types of mailstore - plain files and
maildir. It is expected that additional git-based mailstore will
follow.
This patch contains only the interface changes. No functionality is
added, removed or changed.
A new configuration group [mailstore] is defined. Currently, there is
only one key 'type' whose value determines the used mailstore. The
default value of this option is the plain-file mailstore currently
implemented in notmuch.
Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz>
---
lib/Makefile.local | 2 +
lib/database-private.h | 1 +
lib/database.cc | 15 ++++++--
lib/mailstore-files.c | 43 +++++++++++++++++++++++++
lib/mailstore-private.h | 59 ++++++++++++++++++++++++++++++++++
lib/mailstore.c | 77 ++++++++++++++++++++++++++++++++++++++++++++
lib/message.cc | 13 +++++++
lib/notmuch.h | 81 ++++++++++++++++++++++++++++++++++++++++++++--
notmuch-client.h | 7 ++++
notmuch-config.c | 34 +++++++++++++++++++
notmuch-count.c | 3 +-
notmuch-dump.c | 3 +-
notmuch-new.c | 6 ++-
notmuch-reply.c | 3 +-
notmuch-restore.c | 3 +-
notmuch-search-tags.c | 3 +-
notmuch-search.c | 3 +-
notmuch-show.c | 3 +-
notmuch-tag.c | 3 +-
19 files changed, 344 insertions(+), 18 deletions(-)
create mode 100644 lib/mailstore-files.c
create mode 100644 lib/mailstore-private.h
create mode 100644 lib/mailstore.c
diff --git a/lib/Makefile.local b/lib/Makefile.local
index 495b27e..fc8883d 100644
--- a/lib/Makefile.local
+++ b/lib/Makefile.local
@@ -3,6 +3,8 @@ extra_cflags += -I$(dir)
libnotmuch_c_srcs = \
$(dir)/libsha1.c \
+ $(dir)/mailstore.c \
+ $(dir)/mailstore-files.c\
$(dir)/message-file.c \
$(dir)/messages.c \
$(dir)/sha1.c \
diff --git a/lib/database-private.h b/lib/database-private.h
index 41918d7..4499b1a 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -49,6 +49,7 @@ struct _notmuch_database {
Xapian::TermGenerator *term_gen;
Xapian::ValueRangeProcessor *value_range_processor;
+ notmuch_mailstore_t *mailstore;
};
/* Convert tags from Xapian internal format to notmuch format.
diff --git a/lib/database.cc b/lib/database.cc
index c91e97c..93c8d0f 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -19,6 +19,7 @@
*/
#include "database-private.h"
+#include "mailstore-private.h"
#include <iostream>
@@ -438,7 +439,7 @@ parse_references (void *ctx,
}
notmuch_database_t *
-notmuch_database_create (const char *path)
+notmuch_database_create (const char *path, notmuch_mailstore_t *mailstore)
{
notmuch_database_t *notmuch = NULL;
char *notmuch_path = NULL;
@@ -474,7 +475,8 @@ notmuch_database_create (const char *path)
}
notmuch = notmuch_database_open (path,
- NOTMUCH_DATABASE_MODE_READ_WRITE);
+ NOTMUCH_DATABASE_MODE_READ_WRITE,
+ mailstore);
notmuch_database_upgrade (notmuch, NULL, NULL);
DONE:
@@ -497,7 +499,8 @@ _notmuch_database_ensure_writable (notmuch_database_t *notmuch)
notmuch_database_t *
notmuch_database_open (const char *path,
- notmuch_database_mode_t mode)
+ notmuch_database_mode_t mode,
+ notmuch_mailstore_t *mailstore)
{
notmuch_database_t *notmuch = NULL;
char *notmuch_path = NULL, *xapian_path = NULL;
@@ -605,6 +608,9 @@ notmuch_database_open (const char *path,
prefix_t *prefix = &PROBABILISTIC_PREFIX[i];
notmuch->query_parser->add_prefix (prefix->name, prefix->prefix);
}
+
+ notmuch->mailstore = mailstore;
+ mailstore->notmuch = notmuch;
} catch (const Xapian::Error &error) {
fprintf (stderr, "A Xapian exception occurred opening database: %s\n",
error.get_msg().c_str());
@@ -1493,7 +1499,8 @@ notmuch_database_add_message (notmuch_database_t *notmuch,
DONE:
if (message) {
- if (ret == NOTMUCH_STATUS_SUCCESS && message_ret)
+ if ((ret == NOTMUCH_STATUS_SUCCESS ||
+ ret == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) && message_ret)
*message_ret = message;
else
notmuch_message_destroy (message);
diff --git a/lib/mailstore-files.c b/lib/mailstore-files.c
new file mode 100644
index 0000000..92d7f5d
--- /dev/null
+++ b/lib/mailstore-files.c
@@ -0,0 +1,43 @@
+/* mailstore-files.c - Original notmuch mail store - a collection of
+ * plain-text email messages (one message per file).
+ *
+ * Copyright © 2009 Carl Worth
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see http://www.gnu.org/licenses/ .
+ *
+ * Authors: Carl Worth <cworth@cworth.org>
+ * Michal Sojka <sojkam1@fel.cvut.cz>
+ */
+
+#include "notmuch.h"
+#include "mailstore-private.h"
+
+
+static FILE *
+open_file(notmuch_mailstore_t *mailstore, const char *filename)
+{
+ const char *db_path, *abs_filename;
+
+ db_path = notmuch_database_get_path(mailstore->notmuch);
+ abs_filename = talloc_asprintf(mailstore, "%s/%s", db_path, filename);
+ if (unlikely(abs_filename == NULL))
+ return NULL;
+ return fopen (abs_filename, "r");
+}
+
+/* Original notmuch mail store */
+struct _notmuch_mailstore mailstore_files = {
+ .type = "files",
+ .open_file = open_file,
+};
diff --git a/lib/mailstore-private.h b/lib/mailstore-private.h
new file mode 100644
index 0000000..f606e3f
--- /dev/null
+++ b/lib/mailstore-private.h
@@ -0,0 +1,59 @@
+/* mailstore-private.h - Mailstore abstraction
+ *
+ * Copyright © 2010 Michal Sojka
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see http://www.gnu.org/licenses/ .
+ *
+ * Author: Michal Sojka <sojkam1@fel.cvut.cz>
+ */
+
+#ifndef MAILSTORE_PRIVATE_H
+#define MAILSTORE_PRIVATE_H
+
+#include "notmuch-private.h"
+
+struct _notmuch_mailstore {
+ /* Constant fields */
+ const char *type; /* identification in .notmuch-config */
+ void *priv;
+
+ /* Count files in mailstore so that initial invocation of notmuch
+ * new can report remaining time */
+ void (*count_files)(notmuch_mailstore_t *mailstore,
+ const char *path, int *count,
+ volatile sig_atomic_t *interrupted);
+ /* Scan for new messages and add them to the database. */
+ notmuch_private_status_t (*index_new)(notmuch_mailstore_t *mailstore,
+ const char* path,
+ notmuch_indexing_context_t *ctx);
+ /* Store the tags assigned to the 'messages' to the
+ * 'mailstore'. */
+ notmuch_private_status_t (*sync_tags)(notmuch_mailstore_t *mailstore,
+ notmuch_message_t *message);
+ /* Return file handle from which the message can be read.
+ *
+ * Currently, this is not used and all message access is hardcoded
+ * to notmuch. We will need this operation for git-based and other
+ * non-filesystem mailstores. */
+ FILE * (*open_file)(notmuch_mailstore_t *mailstore,
+ const char *filename);
+
+ /* Run-time fields */
+ notmuch_database_t *notmuch;
+};
+
+extern struct _notmuch_mailstore mailstore_files;
+extern struct _notmuch_mailstore mailstore_maildir;
+
+#endif /* MAILSTORE_PRIVATE_H */
diff --git a/lib/mailstore.c b/lib/mailstore.c
new file mode 100644
index 0000000..cdac9a6
--- /dev/null
+++ b/lib/mailstore.c
@@ -0,0 +1,77 @@
+/* mailstore.c - Mailstore abstraction
+ *
+ * Copyright © 2010 Michal Sojka
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see http://www.gnu.org/licenses/ .
+ *
+ * Author: Michal Sojka <sojkam1@fel.cvut.cz>
+ */
+
+#include <notmuch.h>
+#include "mailstore-private.h"
+
+#define ARRAY_SIZE(arr) (sizeof (arr) / sizeof (arr[0]))
+
+static notmuch_mailstore_t *available[] = {
+ &mailstore_files,
+};
+
+notmuch_mailstore_t *
+notmuch_mailstore_get_by_type (const char *type)
+{
+ unsigned i;
+
+ if (type == NULL)
+ return available[0];
+
+ for (i = 0; i < ARRAY_SIZE(available); i++) {
+ if (strcasecmp (type, available[i]->type) == 0)
+ return available[i];
+ }
+ return NULL;
+}
+
+const char *
+notmuch_mailstore_get_type (notmuch_mailstore_t *mailstore)
+{
+ if (!mailstore)
+ return NULL;
+
+ return mailstore->type;
+}
+
+void
+notmuch_mailstore_count_files (notmuch_mailstore_t *mailstore,
+ const char *path, int *count,
+ volatile sig_atomic_t *interrupted)
+{
+ mailstore->count_files (mailstore, path, count, interrupted);
+}
+
+notmuch_status_t
+notmuch_mailstore_index_new (notmuch_mailstore_t *mailstore,
+ const char *path,
+ notmuch_indexing_context_t *ctx)
+{
+ notmuch_private_status_t status;
+ status = mailstore->index_new (mailstore, path, ctx);
+ return COERCE_STATUS (status, "Index new");
+}
+
+FILE *
+notmuch_mailstore_open_file (notmuch_mailstore_t *mailstore,
+ const char *filename)
+{
+ return mailstore->open_file (mailstore, filename);
+}
diff --git a/lib/message.cc b/lib/message.cc
index 721c9a6..c32ee7d 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -20,6 +20,7 @@
#include "notmuch-private.h"
#include "database-private.h"
+#include "mailstore-private.h"
#include <stdint.h>
@@ -555,10 +556,22 @@ void
_notmuch_message_sync (notmuch_message_t *message)
{
Xapian::WritableDatabase *db;
+ notmuch_private_status_t status;
if (message->notmuch->mode == NOTMUCH_DATABASE_MODE_READ_ONLY)
return;
+ if (message->notmuch->mailstore->sync_tags) {
+ status = message->notmuch->mailstore->sync_tags (message->notmuch->mailstore,
+ message);
+ if (status != NOTMUCH_PRIVATE_STATUS_SUCCESS) {
+ fprintf (stderr, "Error: Cannot sync tags to mailstore (%s)\n",
+ notmuch_status_to_string ((notmuch_status_t)status));
+ /* Exit to avoid unsynchronized mailstore. */
+ exit(1);
+ }
+ }
+
db = static_cast <Xapian::WritableDatabase *> (message->notmuch->xapian_db);
db->replace_document (message->doc_id, message->doc);
}
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 0d9cb0f..8fa7b31 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -32,6 +32,8 @@
NOTMUCH_BEGIN_DECLS
#include <time.h>
+#include <signal.h> /* For sig_atomic_t */
+#include <stdio.h> /* For FILE */
#ifndef FALSE
#define FALSE 0
@@ -119,6 +121,8 @@ typedef struct _notmuch_message notmuch_message_t;
typedef struct _notmuch_tags notmuch_tags_t;
typedef struct _notmuch_directory notmuch_directory_t;
typedef struct _notmuch_filenames notmuch_filenames_t;
+typedef struct _notmuch_mailstore notmuch_mailstore_t;
+
/* Create a new, empty notmuch database located at 'path'.
*
@@ -127,6 +131,8 @@ typedef struct _notmuch_filenames notmuch_filenames_t;
* create a new ".notmuch" directory within 'path' where notmuch will
* store its data.
*
+ * FIXME: Mailstore...
+ *
* After a successful call to notmuch_database_create, the returned
* database will be open so the caller should call
* notmuch_database_close when finished with it.
@@ -140,7 +146,7 @@ typedef struct _notmuch_filenames notmuch_filenames_t;
* an error message on stderr).
*/
notmuch_database_t *
-notmuch_database_create (const char *path);
+notmuch_database_create (const char *path, notmuch_mailstore_t *mailstore);
typedef enum {
NOTMUCH_DATABASE_MODE_READ_ONLY = 0,
@@ -152,6 +158,8 @@ typedef enum {
/* Open an existing notmuch database located at 'path'.
*
+ * The database will index mails stored in 'mailstore'.
+ *
* The database should have been created at some time in the past,
* (not necessarily by this process), by calling
* notmuch_database_create with 'path'. By default the database should be
@@ -169,7 +177,8 @@ typedef enum {
*/
notmuch_database_t *
notmuch_database_open (const char *path,
- notmuch_database_mode_t mode);
+ notmuch_database_mode_t mode,
+ notmuch_mailstore_t *mailstore);
/* Close the given notmuch database, freeing all associated
* resources. See notmuch_database_open. */
@@ -236,7 +245,8 @@ notmuch_database_get_directory (notmuch_database_t *database,
* notmuch database will reference the filename, and will not copy the
* entire contents of the file.
*
- * If 'message' is not NULL, then, on successful return '*message'
+ * If 'message' is not NULL, then, on successful return
+ * (NOTMUCH_STATUS_SUCCESS or NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) '*message'
* will be initialized to a message object that can be used for things
* such as adding tags to the just-added message. The user should call
* notmuch_message_destroy when done with the message. On any failure
@@ -495,7 +505,7 @@ notmuch_threads_destroy (notmuch_threads_t *threads);
*/
unsigned
notmuch_query_count_messages (notmuch_query_t *query);
-
+
/* Get the thread ID of 'thread'.
*
* The returned string belongs to 'thread' and as such, should not be
@@ -1104,6 +1114,69 @@ notmuch_filenames_move_to_next (notmuch_filenames_t *filenames);
void
notmuch_filenames_destroy (notmuch_filenames_t *filenames);
+/* Returns notmuch_mailstore_t object of a given 'type'.
+ *
+ * The 'type' is case insensitive. If 'type' is NULL, a default
+ * mailstore is returned.
+ *
+ * When no available mailstore matches 'type', NULL is returned.
+ */
+notmuch_mailstore_t *
+notmuch_mailstore_get_by_type (const char *type);
+
+/* Returns the type of a mailstore.
+ *
+ * The returned string is owned by mailstore so should not be modified
+ * nor freed by the caller.
+ */
+const char *
+notmuch_mailstore_get_type (notmuch_mailstore_t *mailstore);
+
+/* Count emails in mailstore.
+ *
+ * This function returns when interrupted is set to non-zero value.
+ */
+void
+notmuch_mailstore_count_files (notmuch_mailstore_t *mailstore,
+ const char *path, int *count,
+ volatile sig_atomic_t *interrupted);
+
+typedef struct notmuch_indexing_context {
+ int total_files;
+ int processed_files;
+ int added_messages;
+ int renamed_files;
+ int removed_files;
+
+ void *priv;
+
+ int verbose;
+ volatile sig_atomic_t interrupted;
+ volatile sig_atomic_t print_progress;
+ void (*print_progress_cb)(struct notmuch_indexing_context *ctx);
+ void (*print_verbose_cb)(struct notmuch_indexing_context *ctx, const char *filename);
+} notmuch_indexing_context_t;
+
+
+/* Index new mails in the mailstore.
+ *
+ * Mailstore is responsible for detecting new emails, adding them to
+ * the database and tagging them with tags stored in the mailstore.
+ */
+notmuch_status_t
+notmuch_mailstore_index_new (notmuch_mailstore_t *mailstore,
+ const char *path,
+ notmuch_indexing_context_t *ctx);
+
+/* Returns file handle to the file in the mailstore.
+ *
+ * TODO: This function is currently not used and access to mails is
+ * hardcoded in other parts of notmuch.
+ */
+FILE *
+notmuch_mailstore_open_file (notmuch_mailstore_t *mailstore,
+ const char *filename);
+
NOTMUCH_END_DECLS
#endif
diff --git a/notmuch-client.h b/notmuch-client.h
index c80b39c..4794b8e 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -171,6 +171,13 @@ notmuch_config_set_user_other_email (notmuch_config_t *config,
const char *other_email[],
size_t length);
+notmuch_mailstore_t *
+notmuch_config_get_mailstore(notmuch_config_t *config);
+
+void
+notmuch_config_set_mailstore(notmuch_config_t *config,
+ notmuch_mailstore_t *mailstore);
+
notmuch_bool_t
debugger_is_active (void);
diff --git a/notmuch-config.c b/notmuch-config.c
index 95430db..299423a 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -62,6 +62,7 @@ struct _notmuch_config {
char *user_primary_email;
char **user_other_email;
size_t user_other_email_length;
+ notmuch_mailstore_t *mailstore;
};
static int
@@ -199,6 +200,7 @@ notmuch_config_open (void *ctx,
config->user_primary_email = NULL;
config->user_other_email = NULL;
config->user_other_email_length = 0;
+ config->mailstore = NULL;
if (! g_key_file_load_from_file (config->key_file,
config->filename,
@@ -264,6 +266,10 @@ notmuch_config_open (void *ctx,
}
}
+ if (notmuch_config_get_mailstore (config) == NULL) {
+ notmuch_config_set_mailstore(config, notmuch_mailstore_get_by_type(NULL));
+ }
+
/* When we create a new configuration file here, we add some
* comments to help the user understand what can be done. */
if (is_new) {
@@ -452,3 +458,31 @@ notmuch_config_set_user_other_email (notmuch_config_t *config,
talloc_free (config->user_other_email);
config->user_other_email = NULL;
}
+
+notmuch_mailstore_t *
+notmuch_config_get_mailstore(notmuch_config_t *config)
+{
+ char *type;
+
+ if (config->mailstore == NULL) {
+ type = g_key_file_get_string (config->key_file,
+ "mailstore", "type", NULL);
+ if (type) {
+ config->mailstore = notmuch_mailstore_get_by_type(type);
+ free (type);
+ }
+ }
+
+ return config->mailstore;
+}
+
+void
+notmuch_config_set_mailstore(notmuch_config_t *config,
+ notmuch_mailstore_t *mailstore)
+{
+ g_key_file_set_string (config->key_file,
+ "mailstore", "type",
+ notmuch_mailstore_get_type(mailstore));
+
+ config->mailstore = NULL;
+}
diff --git a/notmuch-count.c b/notmuch-count.c
index 77aa433..631b8f8 100644
--- a/notmuch-count.c
+++ b/notmuch-count.c
@@ -81,7 +81,8 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
return 1;
notmuch = notmuch_database_open (notmuch_config_get_database_path (config),
- NOTMUCH_DATABASE_MODE_READ_ONLY);
+ NOTMUCH_DATABASE_MODE_READ_ONLY,
+ notmuch_config_get_mailstore(config));
if (notmuch == NULL)
return 1;
diff --git a/notmuch-dump.c b/notmuch-dump.c
index 7e7bc17..e74dfcb 100644
--- a/notmuch-dump.c
+++ b/notmuch-dump.c
@@ -36,7 +36,8 @@ notmuch_dump_command (unused (void *ctx), int argc, char *argv[])
return 1;
notmuch = notmuch_database_open (notmuch_config_get_database_path (config),
- NOTMUCH_DATABASE_MODE_READ_ONLY);
+ NOTMUCH_DATABASE_MODE_READ_ONLY,
+ notmuch_config_get_mailstore (config));
if (notmuch == NULL)
return 1;
diff --git a/notmuch-new.c b/notmuch-new.c
index 44b50aa..2d0ba6c 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -749,11 +749,13 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
return 1;
printf ("Found %d total files (that's not much mail).\n", count);
- notmuch = notmuch_database_create (db_path);
+ notmuch = notmuch_database_create (db_path,
+ notmuch_config_get_mailstore (config));
add_files_state.total_files = count;
} else {
notmuch = notmuch_database_open (db_path,
- NOTMUCH_DATABASE_MODE_READ_WRITE);
+ NOTMUCH_DATABASE_MODE_READ_WRITE,
+ notmuch_config_get_mailstore (config));
if (notmuch == NULL)
return 1;
diff --git a/notmuch-reply.c b/notmuch-reply.c
index 6c15536..0d6b7cf 100644
--- a/notmuch-reply.c
+++ b/notmuch-reply.c
@@ -468,7 +468,8 @@ notmuch_reply_command (void *ctx, int argc, char *argv[])
}
notmuch = notmuch_database_open (notmuch_config_get_database_path (config),
- NOTMUCH_DATABASE_MODE_READ_ONLY);
+ NOTMUCH_DATABASE_MODE_READ_ONLY,
+ notmuch_config_get_mailstore (config));
if (notmuch == NULL)
return 1;
diff --git a/notmuch-restore.c b/notmuch-restore.c
index b0a4e1c..97d817b 100644
--- a/notmuch-restore.c
+++ b/notmuch-restore.c
@@ -37,7 +37,8 @@ notmuch_restore_command (unused (void *ctx), int argc, char *argv[])
return 1;
notmuch = notmuch_database_open (notmuch_config_get_database_path (config),
- NOTMUCH_DATABASE_MODE_READ_WRITE);
+ NOTMUCH_DATABASE_MODE_READ_WRITE,
+ notmuch_config_get_mailstore (config));
if (notmuch == NULL)
return 1;
diff --git a/notmuch-search-tags.c b/notmuch-search-tags.c
index 6f3cfcc..323f416 100644
--- a/notmuch-search-tags.c
+++ b/notmuch-search-tags.c
@@ -52,7 +52,8 @@ notmuch_search_tags_command (void *ctx, int argc, char *argv[])
}
db = notmuch_database_open (notmuch_config_get_database_path (config),
- NOTMUCH_DATABASE_MODE_READ_ONLY);
+ NOTMUCH_DATABASE_MODE_READ_ONLY,
+ notmuch_config_get_mailstore (config));
if (db == NULL) {
goto error;
}
diff --git a/notmuch-search.c b/notmuch-search.c
index 4e3514b..d57479d 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -245,7 +245,8 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
return 1;
notmuch = notmuch_database_open (notmuch_config_get_database_path (config),
- NOTMUCH_DATABASE_MODE_READ_ONLY);
+ NOTMUCH_DATABASE_MODE_READ_ONLY,
+ notmuch_config_get_mailstore (config));
if (notmuch == NULL)
return 1;
diff --git a/notmuch-show.c b/notmuch-show.c
index ff1fecb..26581c1 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -458,7 +458,8 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
}
notmuch = notmuch_database_open (notmuch_config_get_database_path (config),
- NOTMUCH_DATABASE_MODE_READ_ONLY);
+ NOTMUCH_DATABASE_MODE_READ_ONLY,
+ notmuch_config_get_mailstore (config));
if (notmuch == NULL)
return 1;
diff --git a/notmuch-tag.c b/notmuch-tag.c
index 8b6f7dc..5e1a5c3 100644
--- a/notmuch-tag.c
+++ b/notmuch-tag.c
@@ -97,7 +97,8 @@ notmuch_tag_command (void *ctx, unused (int argc), unused (char *argv[]))
return 1;
notmuch = notmuch_database_open (notmuch_config_get_database_path (config),
- NOTMUCH_DATABASE_MODE_READ_WRITE);
+ NOTMUCH_DATABASE_MODE_READ_WRITE,
+ notmuch_config_get_mailstore (config));
if (notmuch == NULL)
return 1;
--
1.7.0.2
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [PATCH v2 2/4] Conversion to mailstore abstraction
2010-03-26 21:41 ` Michal Sojka
2010-03-26 21:42 ` [PATCH v2 1/4] Mailstore abstraction interface Michal Sojka
@ 2010-03-26 21:42 ` Michal Sojka
2010-03-26 21:42 ` [PATCH v2 3/4] Add maildir-based mailstore Michal Sojka
2010-03-26 21:42 ` [PATCH v2 4/4] Tests for " Michal Sojka
3 siblings, 0 replies; 25+ messages in thread
From: Michal Sojka @ 2010-03-26 21:42 UTC (permalink / raw)
To: notmuch
The code for detection of new files in the mailstore and their
addition to the database is moved from notmuch-new.c to
lib/mailstore-files.c, where it is called by the abstract mailstore
interface.
The code was only changed to allow the progress reporting functions to
be implemented outside of notmuch library, as can be seen by
git diff HEAD^^ -C
Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz>
---
lib/mailstore-files.c | 581 ++++++++++++++++++++++++++++++++++++++++++++
notmuch-new.c | 640 ++++---------------------------------------------
2 files changed, 634 insertions(+), 587 deletions(-)
diff --git a/lib/mailstore-files.c b/lib/mailstore-files.c
index 92d7f5d..e4086ba 100644
--- a/lib/mailstore-files.c
+++ b/lib/mailstore-files.c
@@ -20,9 +20,587 @@
* Michal Sojka <sojkam1@fel.cvut.cz>
*/
+#define _GNU_SOURCE /* For asprintf() */
#include "notmuch.h"
#include "mailstore-private.h"
+#include <dirent.h>
+typedef struct _filename_node {
+ char *filename;
+ struct _filename_node *next;
+} _filename_node_t;
+
+typedef struct _filename_list {
+ _filename_node_t *head;
+ _filename_node_t **tail;
+} _filename_list_t;
+
+typedef struct _indexing_context_priv {
+ _filename_list_t *removed_files;
+ _filename_list_t *removed_directories;
+} _indexing_context_priv_t;
+
+static _filename_list_t *
+_filename_list_create (const void *ctx)
+{
+ _filename_list_t *list;
+
+ list = talloc (ctx, _filename_list_t);
+ if (list == NULL)
+ return NULL;
+
+ list->head = NULL;
+ list->tail = &list->head;
+
+ return list;
+}
+
+static void
+_filename_list_add (_filename_list_t *list,
+ const char *filename)
+{
+ _filename_node_t *node = talloc (list, _filename_node_t);
+
+ node->filename = talloc_strdup (list, filename);
+ node->next = NULL;
+
+ *(list->tail) = node;
+ list->tail = &node->next;
+}
+
+static void
+tag_inbox_and_unread (notmuch_message_t *message)
+{
+ notmuch_message_add_tag (message, "inbox");
+ notmuch_message_add_tag (message, "unread");
+}
+
+static int
+dirent_sort_inode (const struct dirent **a, const struct dirent **b)
+{
+ return ((*a)->d_ino < (*b)->d_ino) ? -1 : 1;
+}
+
+static int
+dirent_sort_strcmp_name (const struct dirent **a, const struct dirent **b)
+{
+ return strcmp ((*a)->d_name, (*b)->d_name);
+}
+
+/* Test if the directory looks like a Maildir directory.
+ *
+ * Search through the array of directory entries to see if we can find all
+ * three subdirectories typical for Maildir, that is "new", "cur", and "tmp".
+ *
+ * Return 1 if the directory looks like a Maildir and 0 otherwise.
+ */
+static int
+_entries_resemble_maildir (struct dirent **entries, int count)
+{
+ int i, found = 0;
+
+ for (i = 0; i < count; i++) {
+ if (entries[i]->d_type != DT_DIR && entries[i]->d_type != DT_UNKNOWN)
+ continue;
+
+ if (strcmp(entries[i]->d_name, "new") == 0 ||
+ strcmp(entries[i]->d_name, "cur") == 0 ||
+ strcmp(entries[i]->d_name, "tmp") == 0)
+ {
+ found++;
+ if (found == 3)
+ return 1;
+ }
+ }
+
+ return 0;
+}
+
+
+/* Examine 'path' recursively as follows:
+ *
+ * o Ask the filesystem for the mtime of 'path' (fs_mtime)
+ * o Ask the database for its timestamp of 'path' (db_mtime)
+ *
+ * o Ask the filesystem for files and directories within 'path'
+ * (via scandir and stored in fs_entries)
+ * o Ask the database for files and directories within 'path'
+ * (db_files and db_subdirs)
+ *
+ * o Pass 1: For each directory in fs_entries, recursively call into
+ * this same function.
+ *
+ * o Pass 2: If 'fs_mtime' > 'db_mtime', then walk fs_entries
+ * simultaneously with db_files and db_subdirs. Look for one of
+ * three interesting cases:
+ *
+ * 1. Regular file in fs_entries and not in db_files
+ * This is a new file to add_message into the database.
+ *
+ * 2. Filename in db_files not in fs_entries.
+ * This is a file that has been removed from the mail store.
+ *
+ * 3. Directory in db_subdirs not in fs_entries
+ * This is a directory that has been removed from the mail store.
+ *
+ * Note that the addition of a directory is not interesting here,
+ * since that will have been taken care of in pass 1. Also, we
+ * don't immediately act on file/directory removal since we must
+ * ensure that in the case of a rename that the new filename is
+ * added before the old filename is removed, (so that no
+ * information is lost from the database).
+ *
+ * o Tell the database to update its time of 'path' to 'fs_mtime'
+ */
+static notmuch_status_t
+add_files_recursive (notmuch_mailstore_t *mailstore,
+ const char *path,
+ notmuch_indexing_context_t *state)
+{
+ DIR *dir = NULL;
+ struct dirent *entry = NULL;
+ char *next = NULL;
+ time_t fs_mtime, db_mtime;
+ notmuch_status_t status, ret = NOTMUCH_STATUS_SUCCESS;
+ notmuch_message_t *message = NULL;
+ struct dirent **fs_entries = NULL;
+ int i, num_fs_entries;
+ notmuch_directory_t *directory;
+ notmuch_filenames_t *db_files = NULL;
+ notmuch_filenames_t *db_subdirs = NULL;
+ struct stat st;
+ notmuch_bool_t is_maildir, new_directory;
+ _indexing_context_priv_t *priv = state->priv;
+ notmuch_database_t *notmuch = mailstore->notmuch;
+
+ if (stat (path, &st)) {
+ fprintf (stderr, "Error reading directory %s: %s\n",
+ path, strerror (errno));
+ return NOTMUCH_STATUS_FILE_ERROR;
+ }
+
+ /* This is not an error since we may have recursed based on a
+ * symlink to a regular file, not a directory, and we don't know
+ * that until this stat. */
+ if (! S_ISDIR (st.st_mode))
+ return NOTMUCH_STATUS_SUCCESS;
+
+ fs_mtime = st.st_mtime;
+
+ directory = notmuch_database_get_directory (notmuch, path);
+ db_mtime = notmuch_directory_get_mtime (directory);
+
+ if (db_mtime == 0) {
+ new_directory = TRUE;
+ db_files = NULL;
+ db_subdirs = NULL;
+ } else {
+ new_directory = FALSE;
+ db_files = notmuch_directory_get_child_files (directory);
+ db_subdirs = notmuch_directory_get_child_directories (directory);
+ }
+
+ /* If the database knows about this directory, then we sort based
+ * on strcmp to match the database sorting. Otherwise, we can do
+ * inode-based sorting for faster filesystem operation. */
+ num_fs_entries = scandir (path, &fs_entries, 0,
+ new_directory ?
+ dirent_sort_inode : dirent_sort_strcmp_name);
+
+ if (num_fs_entries == -1) {
+ fprintf (stderr, "Error opening directory %s: %s\n",
+ path, strerror (errno));
+ ret = NOTMUCH_STATUS_FILE_ERROR;
+ goto DONE;
+ }
+
+ /* Pass 1: Recurse into all sub-directories. */
+ is_maildir = _entries_resemble_maildir (fs_entries, num_fs_entries);
+
+ for (i = 0; i < num_fs_entries; i++) {
+ if (state->interrupted)
+ break;
+
+ entry = fs_entries[i];
+
+ /* We only want to descend into directories.
+ * But symlinks can be to directories too, of course.
+ *
+ * And if the filesystem doesn't tell us the file type in the
+ * scandir results, then it might be a directory (and if not,
+ * then we'll stat and return immediately in the next level of
+ * recursion). */
+ if (entry->d_type != DT_DIR &&
+ entry->d_type != DT_LNK &&
+ entry->d_type != DT_UNKNOWN)
+ {
+ continue;
+ }
+
+ /* Ignore special directories to avoid infinite recursion.
+ * Also ignore the .notmuch directory and any "tmp" directory
+ * that appears within a maildir.
+ */
+ /* XXX: Eventually we'll want more sophistication to let the
+ * user specify files to be ignored. */
+ if (strcmp (entry->d_name, ".") == 0 ||
+ strcmp (entry->d_name, "..") == 0 ||
+ (is_maildir && strcmp (entry->d_name, "tmp") == 0) ||
+ strcmp (entry->d_name, ".notmuch") ==0)
+ {
+ continue;
+ }
+
+ next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
+ status = add_files_recursive (mailstore, next, state);
+ if (status && ret == NOTMUCH_STATUS_SUCCESS)
+ ret = status;
+ talloc_free (next);
+ next = NULL;
+ }
+
+ /* If this directory hasn't been modified since the last
+ * "notmuch new", then we can skip the second pass entirely. */
+ if (fs_mtime <= db_mtime)
+ goto DONE;
+
+ /* Pass 2: Scan for new files, removed files, and removed directories. */
+ for (i = 0; i < num_fs_entries; i++)
+ {
+ if (state->interrupted)
+ break;
+
+ entry = fs_entries[i];
+
+ /* Check if we've walked past any names in db_files or
+ * db_subdirs. If so, these have been deleted. */
+ while (notmuch_filenames_valid (db_files) &&
+ strcmp (notmuch_filenames_get (db_files), entry->d_name) < 0)
+ {
+ char *absolute = talloc_asprintf (priv->removed_files,
+ "%s/%s", path,
+ notmuch_filenames_get (db_files));
+
+ _filename_list_add (priv->removed_files, absolute);
+
+ notmuch_filenames_move_to_next (db_files);
+ }
+
+ while (notmuch_filenames_valid (db_subdirs) &&
+ strcmp (notmuch_filenames_get (db_subdirs), entry->d_name) <= 0)
+ {
+ const char *filename = notmuch_filenames_get (db_subdirs);
+
+ if (strcmp (filename, entry->d_name) < 0)
+ {
+ char *absolute = talloc_asprintf (priv->removed_directories,
+ "%s/%s", path, filename);
+
+ _filename_list_add (priv->removed_directories, absolute);
+ }
+
+ notmuch_filenames_move_to_next (db_subdirs);
+ }
+
+ /* If we're looking at a symlink, we only want to add it if it
+ * links to a regular file, (and not to a directory, say).
+ *
+ * Similarly, if the file is of unknown type (due to filesytem
+ * limitations), then we also need to look closer.
+ *
+ * In either case, a stat does the trick.
+ */
+ if (entry->d_type == DT_LNK || entry->d_type == DT_UNKNOWN) {
+ int err;
+
+ next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
+ err = stat (next, &st);
+ talloc_free (next);
+ next = NULL;
+
+ /* Don't emit an error for a link pointing nowhere, since
+ * the directory-traversal pass will have already done
+ * that. */
+ if (err)
+ continue;
+
+ if (! S_ISREG (st.st_mode))
+ continue;
+ } else if (entry->d_type != DT_REG) {
+ continue;
+ }
+
+ /* Don't add a file that we've added before. */
+ if (notmuch_filenames_valid (db_files) &&
+ strcmp (notmuch_filenames_get (db_files), entry->d_name) == 0)
+ {
+ notmuch_filenames_move_to_next (db_files);
+ continue;
+ }
+
+ /* We're now looking at a regular file that doesn't yet exist
+ * in the database, so add it. */
+ next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
+
+ state->processed_files++;
+
+ if (state->verbose && state->print_verbose_cb)
+ state->print_verbose_cb(state, next);
+
+ status = notmuch_database_add_message (notmuch, next, &message);
+ switch (status) {
+ /* success */
+ case NOTMUCH_STATUS_SUCCESS:
+ state->added_messages++;
+ tag_inbox_and_unread (message);
+ break;
+ /* Non-fatal issues (go on to next file) */
+ case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
+ /* Stay silent on this one. */
+ break;
+ case NOTMUCH_STATUS_FILE_NOT_EMAIL:
+ fprintf (stderr, "Note: Ignoring non-mail file: %s\n",
+ next);
+ break;
+ /* Fatal issues. Don't process anymore. */
+ case NOTMUCH_STATUS_READ_ONLY_DATABASE:
+ case NOTMUCH_STATUS_XAPIAN_EXCEPTION:
+ case NOTMUCH_STATUS_OUT_OF_MEMORY:
+ fprintf (stderr, "Error: %s. Halting processing.\n",
+ notmuch_status_to_string (status));
+ ret = status;
+ goto DONE;
+ default:
+ case NOTMUCH_STATUS_FILE_ERROR:
+ case NOTMUCH_STATUS_NULL_POINTER:
+ case NOTMUCH_STATUS_TAG_TOO_LONG:
+ case NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW:
+ case NOTMUCH_STATUS_LAST_STATUS:
+ INTERNAL_ERROR ("add_message returned unexpected value: %d", status);
+ goto DONE;
+ }
+
+ if (message) {
+ notmuch_message_destroy (message);
+ message = NULL;
+ }
+
+ if (state->print_progress &&
+ state->print_progress_cb) {
+ state->print_progress = 0;
+ state->print_progress_cb (state);
+ }
+
+ talloc_free (next);
+ next = NULL;
+ }
+
+ if (state->interrupted)
+ goto DONE;
+
+ /* Now that we've walked the whole filesystem list, anything left
+ * over in the database lists has been deleted. */
+ while (notmuch_filenames_valid (db_files))
+ {
+ char *absolute = talloc_asprintf (priv->removed_files,
+ "%s/%s", path,
+ notmuch_filenames_get (db_files));
+
+ _filename_list_add (priv->removed_files, absolute);
+
+ notmuch_filenames_move_to_next (db_files);
+ }
+
+ while (notmuch_filenames_valid (db_subdirs))
+ {
+ char *absolute = talloc_asprintf (priv->removed_directories,
+ "%s/%s", path,
+ notmuch_filenames_get (db_subdirs));
+
+ _filename_list_add (priv->removed_directories, absolute);
+
+ notmuch_filenames_move_to_next (db_subdirs);
+ }
+
+ if (! state->interrupted) {
+ status = notmuch_directory_set_mtime (directory, fs_mtime);
+ if (status && ret == NOTMUCH_STATUS_SUCCESS)
+ ret = status;
+ }
+
+DONE:
+ if (next)
+ talloc_free (next);
+ if (entry)
+ free (entry);
+ if (dir)
+ closedir (dir);
+ if (fs_entries)
+ free (fs_entries);
+ if (db_subdirs)
+ notmuch_filenames_destroy (db_subdirs);
+ if (db_files)
+ notmuch_filenames_destroy (db_files);
+ if (directory)
+ notmuch_directory_destroy (directory);
+
+ return ret;
+}
+
+/* XXX: This should be merged with the add_files function since it
+ * shares a lot of logic with it. */
+/* Recursively count all regular files in path and all sub-directories
+ * of path. The result is added to *count (which should be
+ * initialized to zero by the top-level caller before calling
+ * count_files). */
+static void
+count_files (notmuch_mailstore_t *mailstore,
+ const char *path, int *count,
+ volatile sig_atomic_t *interrupted)
+{
+ struct dirent *entry = NULL;
+ char *next;
+ struct stat st;
+ struct dirent **fs_entries = NULL;
+ int num_fs_entries = scandir (path, &fs_entries, 0, dirent_sort_inode);
+ int i = 0;
+
+ (void)mailstore;
+ if (num_fs_entries == -1) {
+ fprintf (stderr, "Warning: failed to open directory %s: %s\n",
+ path, strerror (errno));
+ goto DONE;
+ }
+
+ while (!*interrupted) {
+ if (i == num_fs_entries)
+ break;
+
+ entry = fs_entries[i++];
+
+ /* Ignore special directories to avoid infinite recursion.
+ * Also ignore the .notmuch directory.
+ */
+ /* XXX: Eventually we'll want more sophistication to let the
+ * user specify files to be ignored. */
+ if (strcmp (entry->d_name, ".") == 0 ||
+ strcmp (entry->d_name, "..") == 0 ||
+ strcmp (entry->d_name, ".notmuch") == 0)
+ {
+ continue;
+ }
+
+ if (asprintf (&next, "%s/%s", path, entry->d_name) == -1) {
+ next = NULL;
+ fprintf (stderr, "Error descending from %s to %s: Out of memory\n",
+ path, entry->d_name);
+ continue;
+ }
+
+ stat (next, &st);
+
+ if (S_ISREG (st.st_mode)) {
+ *count = *count + 1;
+ if (*count % 1000 == 0) {
+ printf ("Found %d files so far.\r", *count);
+ fflush (stdout);
+ }
+ } else if (S_ISDIR (st.st_mode)) {
+ count_files (mailstore, next, count, interrupted);
+ }
+
+ free (next);
+ }
+
+DONE:
+ if (entry)
+ free (entry);
+ if (fs_entries)
+ free (fs_entries);
+}
+
+/* Recursively remove all filenames from the database referring to
+ * 'path' (or to any of its children). */
+static void
+_remove_directory (void *ctx,
+ notmuch_database_t *notmuch,
+ const char *path,
+ int *renamed_files,
+ int *removed_files)
+{
+ notmuch_directory_t *directory;
+ notmuch_filenames_t *files, *subdirs;
+ notmuch_status_t status;
+ char *absolute;
+
+ directory = notmuch_database_get_directory (notmuch, path);
+
+ for (files = notmuch_directory_get_child_files (directory);
+ notmuch_filenames_valid (files);
+ notmuch_filenames_move_to_next (files))
+ {
+ absolute = talloc_asprintf (ctx, "%s/%s", path,
+ notmuch_filenames_get (files));
+ status = notmuch_database_remove_message (notmuch, absolute);
+ if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID)
+ *renamed_files = *renamed_files + 1;
+ else
+ *removed_files = *removed_files + 1;
+ talloc_free (absolute);
+ }
+
+ for (subdirs = notmuch_directory_get_child_directories (directory);
+ notmuch_filenames_valid (subdirs);
+ notmuch_filenames_move_to_next (subdirs))
+ {
+ absolute = talloc_asprintf (ctx, "%s/%s", path,
+ notmuch_filenames_get (subdirs));
+ _remove_directory (ctx, notmuch, absolute, renamed_files, removed_files);
+ talloc_free (absolute);
+ }
+
+ notmuch_directory_destroy (directory);
+}
+
+static notmuch_private_status_t
+index_new(notmuch_mailstore_t *mailstore, const char* path,
+ notmuch_indexing_context_t *indexing_ctx)
+{
+ _indexing_context_priv_t *priv;
+ _filename_node_t *f;
+ notmuch_status_t status, ret;
+ notmuch_database_t *notmuch = mailstore->notmuch;
+
+ priv = talloc(NULL, _indexing_context_priv_t);
+ indexing_ctx->priv = priv;
+ if (priv == NULL)
+ return NOTMUCH_STATUS_OUT_OF_MEMORY;
+
+ priv->removed_files = _filename_list_create (priv);
+ priv->removed_directories = _filename_list_create (priv);
+
+ ret = add_files_recursive(mailstore, path, indexing_ctx);
+
+ indexing_ctx->removed_files = 0;
+ indexing_ctx->renamed_files = 0;
+ for (f = priv->removed_files->head; f; f = f->next) {
+ status = notmuch_database_remove_message (notmuch, f->filename);
+ if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID)
+ indexing_ctx->renamed_files++;
+ else
+ indexing_ctx->removed_files++;
+ }
+
+ for (f = priv->removed_directories->head; f; f = f->next) {
+ _remove_directory (priv, notmuch, f->filename,
+ &indexing_ctx->renamed_files,
+ &indexing_ctx->removed_files);
+ }
+
+ talloc_free(priv);
+
+ return ret;
+}
static FILE *
open_file(notmuch_mailstore_t *mailstore, const char *filename)
@@ -39,5 +617,8 @@ open_file(notmuch_mailstore_t *mailstore, const char *filename)
/* Original notmuch mail store */
struct _notmuch_mailstore mailstore_files = {
.type = "files",
+ .count_files = count_files,
+ .index_new = index_new,
+ .sync_tags = NULL,
.open_file = open_file,
};
diff --git a/notmuch-new.c b/notmuch-new.c
index 2d0ba6c..feb6246 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -22,35 +22,12 @@
#include <unistd.h>
-typedef struct _filename_node {
- char *filename;
- struct _filename_node *next;
-} _filename_node_t;
-
-typedef struct _filename_list {
- _filename_node_t *head;
- _filename_node_t **tail;
-} _filename_list_t;
-
-typedef struct {
- int output_is_a_tty;
- int verbose;
-
- int total_files;
- int processed_files;
- int added_messages;
- struct timeval tv_start;
-
- _filename_list_t *removed_files;
- _filename_list_t *removed_directories;
-} add_files_state_t;
-
-static volatile sig_atomic_t do_add_files_print_progress = 0;
+notmuch_indexing_context_t *indexing_ctx;
static void
handle_sigalrm (unused (int signal))
{
- do_add_files_print_progress = 1;
+ indexing_ctx->print_progress = 1;
}
static volatile sig_atomic_t interrupted;
@@ -62,53 +39,24 @@ handle_sigint (unused (int sig))
static char msg[] = "Stopping... \n";
ignored = write(2, msg, sizeof(msg)-1);
- interrupted = 1;
+ indexing_ctx->interrupted = 1;
}
-static _filename_list_t *
-_filename_list_create (const void *ctx)
-{
- _filename_list_t *list;
-
- list = talloc (ctx, _filename_list_t);
- if (list == NULL)
- return NULL;
-
- list->head = NULL;
- list->tail = &list->head;
-
- return list;
-}
-
-static void
-_filename_list_add (_filename_list_t *list,
- const char *filename)
-{
- _filename_node_t *node = talloc (list, _filename_node_t);
-
- node->filename = talloc_strdup (list, filename);
- node->next = NULL;
-
- *(list->tail) = node;
- list->tail = &node->next;
-}
-
-static void
-tag_inbox_and_unread (notmuch_message_t *message)
-{
- notmuch_message_add_tag (message, "inbox");
- notmuch_message_add_tag (message, "unread");
-}
+struct priv_ctx {
+ struct timeval tv_start;
+ int output_is_a_tty;
+};
static void
-add_files_print_progress (add_files_state_t *state)
+add_files_print_progress (notmuch_indexing_context_t *state)
{
+ struct priv_ctx *state2 = state->priv;
struct timeval tv_now;
double elapsed_overall, rate_overall;
gettimeofday (&tv_now, NULL);
- elapsed_overall = notmuch_time_elapsed (state->tv_start, tv_now);
+ elapsed_overall = notmuch_time_elapsed (state2->tv_start, tv_now);
rate_overall = (state->processed_files) / elapsed_overall;
printf ("Processed %d", state->processed_files);
@@ -128,396 +76,42 @@ add_files_print_progress (add_files_state_t *state)
fflush (stdout);
}
-static int
-dirent_sort_inode (const struct dirent **a, const struct dirent **b)
-{
- return ((*a)->d_ino < (*b)->d_ino) ? -1 : 1;
-}
-
-static int
-dirent_sort_strcmp_name (const struct dirent **a, const struct dirent **b)
-{
- return strcmp ((*a)->d_name, (*b)->d_name);
-}
-
-/* Test if the directory looks like a Maildir directory.
- *
- * Search through the array of directory entries to see if we can find all
- * three subdirectories typical for Maildir, that is "new", "cur", and "tmp".
- *
- * Return 1 if the directory looks like a Maildir and 0 otherwise.
- */
-static int
-_entries_resemble_maildir (struct dirent **entries, int count)
-{
- int i, found = 0;
-
- for (i = 0; i < count; i++) {
- if (entries[i]->d_type != DT_DIR && entries[i]->d_type != DT_UNKNOWN)
- continue;
-
- if (strcmp(entries[i]->d_name, "new") == 0 ||
- strcmp(entries[i]->d_name, "cur") == 0 ||
- strcmp(entries[i]->d_name, "tmp") == 0)
- {
- found++;
- if (found == 3)
- return 1;
- }
- }
-
- return 0;
-}
-
-/* Examine 'path' recursively as follows:
- *
- * o Ask the filesystem for the mtime of 'path' (fs_mtime)
- * o Ask the database for its timestamp of 'path' (db_mtime)
- *
- * o Ask the filesystem for files and directories within 'path'
- * (via scandir and stored in fs_entries)
- * o Ask the database for files and directories within 'path'
- * (db_files and db_subdirs)
- *
- * o Pass 1: For each directory in fs_entries, recursively call into
- * this same function.
- *
- * o Pass 2: If 'fs_mtime' > 'db_mtime', then walk fs_entries
- * simultaneously with db_files and db_subdirs. Look for one of
- * three interesting cases:
- *
- * 1. Regular file in fs_entries and not in db_files
- * This is a new file to add_message into the database.
- *
- * 2. Filename in db_files not in fs_entries.
- * This is a file that has been removed from the mail store.
- *
- * 3. Directory in db_subdirs not in fs_entries
- * This is a directory that has been removed from the mail store.
- *
- * Note that the addition of a directory is not interesting here,
- * since that will have been taken care of in pass 1. Also, we
- * don't immediately act on file/directory removal since we must
- * ensure that in the case of a rename that the new filename is
- * added before the old filename is removed, (so that no
- * information is lost from the database).
- *
- * o Tell the database to update its time of 'path' to 'fs_mtime'
- */
-static notmuch_status_t
-add_files_recursive (notmuch_database_t *notmuch,
- const char *path,
- add_files_state_t *state)
+static void
+add_files_print_verbose (notmuch_indexing_context_t *state, const char *filename)
{
- DIR *dir = NULL;
- struct dirent *entry = NULL;
- char *next = NULL;
- time_t fs_mtime, db_mtime;
- notmuch_status_t status, ret = NOTMUCH_STATUS_SUCCESS;
- notmuch_message_t *message = NULL;
- struct dirent **fs_entries = NULL;
- int i, num_fs_entries;
- notmuch_directory_t *directory;
- notmuch_filenames_t *db_files = NULL;
- notmuch_filenames_t *db_subdirs = NULL;
- struct stat st;
- notmuch_bool_t is_maildir, new_directory;
-
- if (stat (path, &st)) {
- fprintf (stderr, "Error reading directory %s: %s\n",
- path, strerror (errno));
- return NOTMUCH_STATUS_FILE_ERROR;
- }
-
- /* This is not an error since we may have recursed based on a
- * symlink to a regular file, not a directory, and we don't know
- * that until this stat. */
- if (! S_ISDIR (st.st_mode))
- return NOTMUCH_STATUS_SUCCESS;
-
- fs_mtime = st.st_mtime;
-
- directory = notmuch_database_get_directory (notmuch, path);
- db_mtime = notmuch_directory_get_mtime (directory);
-
- if (db_mtime == 0) {
- new_directory = TRUE;
- db_files = NULL;
- db_subdirs = NULL;
- } else {
- new_directory = FALSE;
- db_files = notmuch_directory_get_child_files (directory);
- db_subdirs = notmuch_directory_get_child_directories (directory);
- }
-
- /* If the database knows about this directory, then we sort based
- * on strcmp to match the database sorting. Otherwise, we can do
- * inode-based sorting for faster filesystem operation. */
- num_fs_entries = scandir (path, &fs_entries, 0,
- new_directory ?
- dirent_sort_inode : dirent_sort_strcmp_name);
-
- if (num_fs_entries == -1) {
- fprintf (stderr, "Error opening directory %s: %s\n",
- path, strerror (errno));
- ret = NOTMUCH_STATUS_FILE_ERROR;
- goto DONE;
- }
-
- /* Pass 1: Recurse into all sub-directories. */
- is_maildir = _entries_resemble_maildir (fs_entries, num_fs_entries);
-
- for (i = 0; i < num_fs_entries; i++) {
- if (interrupted)
- break;
-
- entry = fs_entries[i];
-
- /* We only want to descend into directories.
- * But symlinks can be to directories too, of course.
- *
- * And if the filesystem doesn't tell us the file type in the
- * scandir results, then it might be a directory (and if not,
- * then we'll stat and return immediately in the next level of
- * recursion). */
- if (entry->d_type != DT_DIR &&
- entry->d_type != DT_LNK &&
- entry->d_type != DT_UNKNOWN)
- {
- continue;
- }
-
- /* Ignore special directories to avoid infinite recursion.
- * Also ignore the .notmuch directory and any "tmp" directory
- * that appears within a maildir.
- */
- /* XXX: Eventually we'll want more sophistication to let the
- * user specify files to be ignored. */
- if (strcmp (entry->d_name, ".") == 0 ||
- strcmp (entry->d_name, "..") == 0 ||
- (is_maildir && strcmp (entry->d_name, "tmp") == 0) ||
- strcmp (entry->d_name, ".notmuch") ==0)
- {
- continue;
- }
-
- next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
- status = add_files_recursive (notmuch, next, state);
- if (status && ret == NOTMUCH_STATUS_SUCCESS)
- ret = status;
- talloc_free (next);
- next = NULL;
- }
-
- /* If this directory hasn't been modified since the last
- * "notmuch new", then we can skip the second pass entirely. */
- if (fs_mtime <= db_mtime)
- goto DONE;
-
- /* Pass 2: Scan for new files, removed files, and removed directories. */
- for (i = 0; i < num_fs_entries; i++)
- {
- if (interrupted)
- break;
+ struct priv_ctx *state2 = state->priv;
+ if (state2->output_is_a_tty)
+ printf("\r\033[K");
- entry = fs_entries[i];
+ printf ("%i/%i: %s",
+ state->processed_files,
+ state->total_files,
+ filename);
- /* Check if we've walked past any names in db_files or
- * db_subdirs. If so, these have been deleted. */
- while (notmuch_filenames_valid (db_files) &&
- strcmp (notmuch_filenames_get (db_files), entry->d_name) < 0)
- {
- char *absolute = talloc_asprintf (state->removed_files,
- "%s/%s", path,
- notmuch_filenames_get (db_files));
-
- _filename_list_add (state->removed_files, absolute);
-
- notmuch_filenames_move_to_next (db_files);
- }
-
- while (notmuch_filenames_valid (db_subdirs) &&
- strcmp (notmuch_filenames_get (db_subdirs), entry->d_name) <= 0)
- {
- const char *filename = notmuch_filenames_get (db_subdirs);
-
- if (strcmp (filename, entry->d_name) < 0)
- {
- char *absolute = talloc_asprintf (state->removed_directories,
- "%s/%s", path, filename);
-
- _filename_list_add (state->removed_directories, absolute);
- }
-
- notmuch_filenames_move_to_next (db_subdirs);
- }
-
- /* If we're looking at a symlink, we only want to add it if it
- * links to a regular file, (and not to a directory, say).
- *
- * Similarly, if the file is of unknown type (due to filesytem
- * limitations), then we also need to look closer.
- *
- * In either case, a stat does the trick.
- */
- if (entry->d_type == DT_LNK || entry->d_type == DT_UNKNOWN) {
- int err;
-
- next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
- err = stat (next, &st);
- talloc_free (next);
- next = NULL;
-
- /* Don't emit an error for a link pointing nowhere, since
- * the directory-traversal pass will have already done
- * that. */
- if (err)
- continue;
-
- if (! S_ISREG (st.st_mode))
- continue;
- } else if (entry->d_type != DT_REG) {
- continue;
- }
-
- /* Don't add a file that we've added before. */
- if (notmuch_filenames_valid (db_files) &&
- strcmp (notmuch_filenames_get (db_files), entry->d_name) == 0)
- {
- notmuch_filenames_move_to_next (db_files);
- continue;
- }
-
- /* We're now looking at a regular file that doesn't yet exist
- * in the database, so add it. */
- next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
-
- state->processed_files++;
-
- if (state->verbose) {
- if (state->output_is_a_tty)
- printf("\r\033[K");
-
- printf ("%i/%i: %s",
- state->processed_files,
- state->total_files,
- next);
-
- putchar((state->output_is_a_tty) ? '\r' : '\n');
- fflush (stdout);
- }
-
- status = notmuch_database_add_message (notmuch, next, &message);
- switch (status) {
- /* success */
- case NOTMUCH_STATUS_SUCCESS:
- state->added_messages++;
- tag_inbox_and_unread (message);
- break;
- /* Non-fatal issues (go on to next file) */
- case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
- /* Stay silent on this one. */
- break;
- case NOTMUCH_STATUS_FILE_NOT_EMAIL:
- fprintf (stderr, "Note: Ignoring non-mail file: %s\n",
- next);
- break;
- /* Fatal issues. Don't process anymore. */
- case NOTMUCH_STATUS_READ_ONLY_DATABASE:
- case NOTMUCH_STATUS_XAPIAN_EXCEPTION:
- case NOTMUCH_STATUS_OUT_OF_MEMORY:
- fprintf (stderr, "Error: %s. Halting processing.\n",
- notmuch_status_to_string (status));
- ret = status;
- goto DONE;
- default:
- case NOTMUCH_STATUS_FILE_ERROR:
- case NOTMUCH_STATUS_NULL_POINTER:
- case NOTMUCH_STATUS_TAG_TOO_LONG:
- case NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW:
- case NOTMUCH_STATUS_LAST_STATUS:
- INTERNAL_ERROR ("add_message returned unexpected value: %d", status);
- goto DONE;
- }
-
- if (message) {
- notmuch_message_destroy (message);
- message = NULL;
- }
-
- if (do_add_files_print_progress) {
- do_add_files_print_progress = 0;
- add_files_print_progress (state);
- }
-
- talloc_free (next);
- next = NULL;
- }
-
- /* Now that we've walked the whole filesystem list, anything left
- * over in the database lists has been deleted. */
- while (notmuch_filenames_valid (db_files))
- {
- char *absolute = talloc_asprintf (state->removed_files,
- "%s/%s", path,
- notmuch_filenames_get (db_files));
-
- _filename_list_add (state->removed_files, absolute);
-
- notmuch_filenames_move_to_next (db_files);
- }
-
- while (notmuch_filenames_valid (db_subdirs))
- {
- char *absolute = talloc_asprintf (state->removed_directories,
- "%s/%s", path,
- notmuch_filenames_get (db_subdirs));
-
- _filename_list_add (state->removed_directories, absolute);
-
- notmuch_filenames_move_to_next (db_subdirs);
- }
-
- if (! interrupted) {
- status = notmuch_directory_set_mtime (directory, fs_mtime);
- if (status && ret == NOTMUCH_STATUS_SUCCESS)
- ret = status;
- }
-
- DONE:
- if (next)
- talloc_free (next);
- if (entry)
- free (entry);
- if (dir)
- closedir (dir);
- if (fs_entries)
- free (fs_entries);
- if (db_subdirs)
- notmuch_filenames_destroy (db_subdirs);
- if (db_files)
- notmuch_filenames_destroy (db_files);
- if (directory)
- notmuch_directory_destroy (directory);
-
- return ret;
+ putchar((state2->output_is_a_tty) ? '\r' : '\n');
+ fflush (stdout);
}
/* This is the top-level entry point for add_files. It does a couple
* of error checks, sets up the progress-printing timer and then calls
* into the recursive function. */
static notmuch_status_t
-add_files (notmuch_database_t *notmuch,
+add_files (notmuch_mailstore_t *mailstore,
const char *path,
- add_files_state_t *state)
+ notmuch_indexing_context_t *state) /* FIXME: rename */
{
+ struct priv_ctx *state2 = state->priv;
notmuch_status_t status;
struct sigaction action;
struct itimerval timerval;
notmuch_bool_t timer_is_active = FALSE;
struct stat st;
- if (state->output_is_a_tty && ! debugger_is_active () && ! state->verbose) {
+ state->print_progress = 0;
+ state->print_progress_cb = add_files_print_progress;
+ state->print_verbose_cb = add_files_print_verbose;
+
+ if (state2->output_is_a_tty && ! debugger_is_active () && ! state->verbose) {
/* Setup our handler for SIGALRM */
memset (&action, 0, sizeof (struct sigaction));
action.sa_handler = handle_sigalrm;
@@ -546,7 +140,7 @@ add_files (notmuch_database_t *notmuch,
return NOTMUCH_STATUS_FILE_ERROR;
}
- status = add_files_recursive (notmuch, path, state);
+ status = notmuch_mailstore_index_new (mailstore, path, state);
if (timer_is_active) {
/* Now stop the timer. */
@@ -564,80 +158,12 @@ add_files (notmuch_database_t *notmuch,
return status;
}
-/* XXX: This should be merged with the add_files function since it
- * shares a lot of logic with it. */
-/* Recursively count all regular files in path and all sub-directories
- * of path. The result is added to *count (which should be
- * initialized to zero by the top-level caller before calling
- * count_files). */
-static void
-count_files (const char *path, int *count)
-{
- struct dirent *entry = NULL;
- char *next;
- struct stat st;
- struct dirent **fs_entries = NULL;
- int num_fs_entries = scandir (path, &fs_entries, 0, dirent_sort_inode);
- int i = 0;
-
- if (num_fs_entries == -1) {
- fprintf (stderr, "Warning: failed to open directory %s: %s\n",
- path, strerror (errno));
- goto DONE;
- }
-
- while (!interrupted) {
- if (i == num_fs_entries)
- break;
-
- entry = fs_entries[i++];
-
- /* Ignore special directories to avoid infinite recursion.
- * Also ignore the .notmuch directory.
- */
- /* XXX: Eventually we'll want more sophistication to let the
- * user specify files to be ignored. */
- if (strcmp (entry->d_name, ".") == 0 ||
- strcmp (entry->d_name, "..") == 0 ||
- strcmp (entry->d_name, ".notmuch") == 0)
- {
- continue;
- }
-
- if (asprintf (&next, "%s/%s", path, entry->d_name) == -1) {
- next = NULL;
- fprintf (stderr, "Error descending from %s to %s: Out of memory\n",
- path, entry->d_name);
- continue;
- }
-
- stat (next, &st);
-
- if (S_ISREG (st.st_mode)) {
- *count = *count + 1;
- if (*count % 1000 == 0) {
- printf ("Found %d files so far.\r", *count);
- fflush (stdout);
- }
- } else if (S_ISDIR (st.st_mode)) {
- count_files (next, count);
- }
-
- free (next);
- }
-
- DONE:
- if (entry)
- free (entry);
- if (fs_entries)
- free (fs_entries);
-}
-
static void
upgrade_print_progress (void *closure,
double progress)
{
- add_files_state_t *state = closure;
+ notmuch_indexing_context_t *state = closure;
+ struct priv_ctx *state2 = state->priv;
printf ("Upgrading database: %.2f%% complete", progress * 100.0);
@@ -647,7 +173,7 @@ upgrade_print_progress (void *closure,
gettimeofday (&tv_now, NULL);
- elapsed = notmuch_time_elapsed (state->tv_start, tv_now);
+ elapsed = notmuch_time_elapsed (state2->tv_start, tv_now);
time_remaining = (elapsed / progress) * (1.0 - progress);
printf (" (");
notmuch_time_print_formatted_seconds (time_remaining);
@@ -659,55 +185,13 @@ upgrade_print_progress (void *closure,
fflush (stdout);
}
-/* Recursively remove all filenames from the database referring to
- * 'path' (or to any of its children). */
-static void
-_remove_directory (void *ctx,
- notmuch_database_t *notmuch,
- const char *path,
- int *renamed_files,
- int *removed_files)
-{
- notmuch_directory_t *directory;
- notmuch_filenames_t *files, *subdirs;
- notmuch_status_t status;
- char *absolute;
-
- directory = notmuch_database_get_directory (notmuch, path);
-
- for (files = notmuch_directory_get_child_files (directory);
- notmuch_filenames_valid (files);
- notmuch_filenames_move_to_next (files))
- {
- absolute = talloc_asprintf (ctx, "%s/%s", path,
- notmuch_filenames_get (files));
- status = notmuch_database_remove_message (notmuch, absolute);
- if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID)
- *renamed_files = *renamed_files + 1;
- else
- *removed_files = *removed_files + 1;
- talloc_free (absolute);
- }
-
- for (subdirs = notmuch_directory_get_child_directories (directory);
- notmuch_filenames_valid (subdirs);
- notmuch_filenames_move_to_next (subdirs))
- {
- absolute = talloc_asprintf (ctx, "%s/%s", path,
- notmuch_filenames_get (subdirs));
- _remove_directory (ctx, notmuch, absolute, renamed_files, removed_files);
- talloc_free (absolute);
- }
-
- notmuch_directory_destroy (directory);
-}
-
int
notmuch_new_command (void *ctx, int argc, char *argv[])
{
notmuch_config_t *config;
notmuch_database_t *notmuch;
- add_files_state_t add_files_state;
+ struct priv_ctx state2;
+ notmuch_indexing_context_t add_files_state; /* FIXME: Rename */
double elapsed;
struct timeval tv_now;
int ret = 0;
@@ -715,13 +199,14 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
const char *db_path;
char *dot_notmuch_path;
struct sigaction action;
- _filename_node_t *f;
- int renamed_files, removed_files;
- notmuch_status_t status;
int i;
+ notmuch_mailstore_t *mailstore;
+
+ indexing_ctx = &add_files_state;
add_files_state.verbose = 0;
- add_files_state.output_is_a_tty = isatty (fileno (stdout));
+ add_files_state.priv = &state2;
+ state2.output_is_a_tty = isatty (fileno (stdout));
for (i = 0; i < argc && argv[i][0] == '-'; i++) {
if (STRNCMP_LITERAL (argv[i], "--verbose") == 0) {
@@ -737,6 +222,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
return 1;
db_path = notmuch_config_get_database_path (config);
+ mailstore = notmuch_config_get_mailstore (config);
dot_notmuch_path = talloc_asprintf (ctx, "%s/%s", db_path, ".notmuch");
@@ -744,7 +230,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
int count;
count = 0;
- count_files (db_path, &count);
+ notmuch_mailstore_count_files (mailstore, db_path, &count, &interrupted);
if (interrupted)
return 1;
@@ -761,7 +247,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
if (notmuch_database_needs_upgrade (notmuch)) {
printf ("Welcome to a new version of notmuch! Your database will now be upgraded.\n");
- gettimeofday (&add_files_state.tv_start, NULL);
+ gettimeofday (&state2.tv_start, NULL);
notmuch_database_upgrade (notmuch, upgrade_print_progress,
&add_files_state);
printf ("Your notmuch database has now been upgraded to database format version %u.\n",
@@ -777,6 +263,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
/* Setup our handler for SIGINT. We do this after having
* potentially done a database upgrade we this interrupt handler
* won't support. */
+ add_files_state.interrupted = 0;
memset (&action, 0, sizeof (struct sigaction));
action.sa_handler = handle_sigint;
sigemptyset (&action.sa_mask);
@@ -788,33 +275,12 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
add_files_state.processed_files = 0;
add_files_state.added_messages = 0;
- gettimeofday (&add_files_state.tv_start, NULL);
-
- add_files_state.removed_files = _filename_list_create (ctx);
- add_files_state.removed_directories = _filename_list_create (ctx);
-
- ret = add_files (notmuch, db_path, &add_files_state);
-
- removed_files = 0;
- renamed_files = 0;
- for (f = add_files_state.removed_files->head; f; f = f->next) {
- status = notmuch_database_remove_message (notmuch, f->filename);
- if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID)
- renamed_files++;
- else
- removed_files++;
- }
-
- for (f = add_files_state.removed_directories->head; f; f = f->next) {
- _remove_directory (ctx, notmuch, f->filename,
- &renamed_files, &removed_files);
- }
+ gettimeofday (&state2.tv_start, NULL);
- talloc_free (add_files_state.removed_files);
- talloc_free (add_files_state.removed_directories);
+ ret = add_files (mailstore, db_path, &add_files_state);
gettimeofday (&tv_now, NULL);
- elapsed = notmuch_time_elapsed (add_files_state.tv_start,
+ elapsed = notmuch_time_elapsed (state2.tv_start,
tv_now);
if (add_files_state.processed_files) {
@@ -839,16 +305,16 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
printf ("No new mail.");
}
- if (removed_files) {
+ if (add_files_state.removed_files) {
printf (" Removed %d %s.",
- removed_files,
- removed_files == 1 ? "message" : "messages");
+ add_files_state.removed_files,
+ add_files_state.removed_files == 1 ? "message" : "messages");
}
- if (renamed_files) {
+ if (add_files_state.renamed_files) {
printf (" Detected %d file %s.",
- renamed_files,
- renamed_files == 1 ? "rename" : "renames");
+ add_files_state.renamed_files,
+ add_files_state.renamed_files == 1 ? "rename" : "renames");
}
printf ("\n");
--
1.7.0.2
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [PATCH v2 3/4] Add maildir-based mailstore
2010-03-26 21:41 ` Michal Sojka
2010-03-26 21:42 ` [PATCH v2 1/4] Mailstore abstraction interface Michal Sojka
2010-03-26 21:42 ` [PATCH v2 2/4] Conversion to mailstore abstraction Michal Sojka
@ 2010-03-26 21:42 ` Michal Sojka
2010-03-26 21:42 ` [PATCH v2 4/4] Tests for " Michal Sojka
3 siblings, 0 replies; 25+ messages in thread
From: Michal Sojka @ 2010-03-26 21:42 UTC (permalink / raw)
To: notmuch
This mailstore allows bi-directional synchronization between maildir
flags and certain tags. The flag-to-tag mapping is defined by flag2tag
array.
The synchronization works this way:
1) Whenever notmuch new is executed, the following happens:
o New messages are tagged with inbox.
o New messages without maildir info in the file name (typically
files in new/) are tagged with unread.
o For new or renamed messages with maildir info present in the
file name, the tags defined in flag2tag are either added or
removed depending on the flags from the file name.
2) Whenever notmuch tag is executed, a new set of flags based on the
tags is constructed for every message and a new file name is
prepared based on the old file name but with the new flags. If the
old message file was in 'new' directory then this is replaced with
'cur' in the new file name. If the new and old file names differ,
the file is renamed and notmuch database is updated accordingly.
The rename happens before the database is updated. In case of crash
between rename and database update, the next run of notmuch new
brings the database in sync with the mail store again.
This mailstore is enabled by putting the following to your
.notmuch-config:
[mailstore]
type=maildir
Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz>
---
lib/database.cc | 7 ++
lib/mailstore-files.c | 203 ++++++++++++++++++++++++++++++++++++++++++++++++-
lib/mailstore.c | 1 +
lib/message.cc | 41 ++++++++++-
lib/notmuch-private.h | 4 +
lib/notmuch.h | 1 +
6 files changed, 253 insertions(+), 4 deletions(-)
diff --git a/lib/database.cc b/lib/database.cc
index 93c8d0f..33ef889 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1471,6 +1471,13 @@ notmuch_database_add_message (notmuch_database_t *notmuch,
_notmuch_message_add_filename (message, filename);
+ /* This is a new message or it has a new filename and as such,
+ * its tags in database either do not exists or might be out
+ * of date. Mailstore assigns the tags later in index_new(),
+ * but until then we should not synchronize the tags back to
+ * the mailstore. */
+ notmuch_message_set_flag(message, NOTMUCH_MESSAGE_FLAG_TAGS_INVALID, TRUE);
+
/* Is this a newly created message object? */
if (private_status == NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND) {
_notmuch_message_add_term (message, "type", "mail");
diff --git a/lib/mailstore-files.c b/lib/mailstore-files.c
index e4086ba..b4e6e42 100644
--- a/lib/mailstore-files.c
+++ b/lib/mailstore-files.c
@@ -24,6 +24,30 @@
#include "notmuch.h"
#include "mailstore-private.h"
#include <dirent.h>
+#include <stdbool.h>
+
+#define ARRAY_SIZE(arr) (sizeof (arr) / sizeof (arr[0]))
+
+struct mailstore_priv {
+ void (*tag_new)(notmuch_message_t *message, const char *filename);
+ int (*tag_renamed)(notmuch_message_t *message, const char *filename);
+};
+
+struct maildir_flag_tag {
+ char flag;
+ const char *tag;
+ bool inverse;
+};
+
+/* ASCII ordered table of Maildir flags and assiciated tags */
+struct maildir_flag_tag flag2tag[] = {
+ { 'D', "draft", false},
+ { 'F', "flagged", false},
+ { 'P', "passed", false},
+ { 'R', "replied", false},
+ { 'S', "unread", true },
+ { 'T', "delete", false},
+};
typedef struct _filename_node {
char *filename;
@@ -69,8 +93,9 @@ _filename_list_add (_filename_list_t *list,
}
static void
-tag_inbox_and_unread (notmuch_message_t *message)
+tag_inbox_and_unread (notmuch_message_t *message, const char *filename)
{
+ (void)filename;
notmuch_message_add_tag (message, "inbox");
notmuch_message_add_tag (message, "unread");
}
@@ -116,6 +141,156 @@ _entries_resemble_maildir (struct dirent **entries, int count)
return 0;
}
+static int
+tag_from_maildir_flags (notmuch_message_t *message, const char *filename)
+{
+ const char *flags, *p;
+ char f;
+ bool valid, unread;
+ unsigned i;
+
+ flags = strstr(filename, ":2,");
+ if (!flags)
+ return -1;
+ flags += 3;
+
+ /* Check that the letters are valid Maildir flags */
+ f = 0;
+ valid = true;
+ for (p=flags; valid && *p; p++) {
+ switch (*p) {
+ case 'P':
+ case 'R':
+ case 'S':
+ case 'T':
+ case 'D':
+ case 'F':
+ if (*p > f) f=*p;
+ else valid = false;
+ break;
+ default:
+ valid = false;
+ }
+ }
+ if (!valid) {
+ fprintf (stderr, "Warning: Invalid maildir flags in filename %s\n", filename);
+ return -1;
+ }
+
+ notmuch_message_freeze(message);
+ unread = true;
+ for (i = 0; i < ARRAY_SIZE(flag2tag); i++) {
+ if ((strchr(flags, flag2tag[i].flag) != NULL) ^ flag2tag[i].inverse) {
+ notmuch_message_add_tag (message, flag2tag[i].tag);
+ } else {
+ notmuch_message_remove_tag (message, flag2tag[i].tag);
+ }
+ }
+ notmuch_message_thaw(message);
+
+ /* From now on, we can synchronize the tags from the database to
+ * the mailstore. */
+ notmuch_message_set_flag(message, NOTMUCH_MESSAGE_FLAG_TAGS_INVALID, FALSE);
+ return 0;
+}
+
+static void
+get_new_flags(notmuch_message_t *message, char *flags)
+{
+ notmuch_tags_t *tags;
+ const char *tag;
+ unsigned i;
+ char *p;
+
+ for (i = 0; i < ARRAY_SIZE(flag2tag); i++)
+ flags[i] = flag2tag[i].inverse ? flag2tag[i].flag : '\0';
+
+ for (tags = notmuch_message_get_tags (message);
+ notmuch_tags_valid (tags);
+ notmuch_tags_move_to_next (tags))
+ {
+ tag = notmuch_tags_get (tags);
+ for (i = 0; i < ARRAY_SIZE(flag2tag); i++) {
+ if (strcmp(tag, flag2tag[i].tag) == 0)
+ flags[i] = flag2tag[i].inverse ? '\0' : flag2tag[i].flag;
+ }
+ }
+
+ p = flags;
+ for (i = 0; i < ARRAY_SIZE(flag2tag); i++) {
+ if (flags[i])
+ *p++ = flags[i];
+ }
+ *p = '\0';
+}
+
+static char *
+get_subdir (char *filename)
+{
+ char *p, *subdir = NULL;
+
+ p = filename + strlen (filename) - 1;
+ while (p > filename + 3 && *p != '/')
+ p--;
+ if (*p == '/') {
+ subdir = p - 3;
+ if (subdir > filename && *(subdir - 1) != '/')
+ subdir = NULL;
+ }
+ return subdir;
+}
+
+/* Store maildir-related tags as maildir flags */
+static notmuch_private_status_t
+maildir_sync_tags (notmuch_mailstore_t *mailstore,
+ notmuch_message_t *message)
+{
+ char flags[ARRAY_SIZE(flag2tag)+1];
+ const char *filename, *p;
+ char *filename_new, *subdir = NULL;
+ int ret;
+
+ (void)mailstore;
+ get_new_flags (message, flags);
+
+ filename = notmuch_message_get_filename (message);
+ p = strstr(filename, ":2,");
+ if (!p)
+ p = filename + strlen(filename);
+
+ filename_new = talloc_size(message, (p-filename) + 3 + sizeof(flags));
+ if (unlikely (filename_new == NULL))
+ return NOTMUCH_STATUS_OUT_OF_MEMORY;
+ memcpy(filename_new, filename, p-filename);
+ filename_new[p-filename] = '\0';
+
+ /* If message is in new/ move it under cur/. */
+ subdir = get_subdir (filename_new);
+ if (subdir && memcmp (subdir, "new/", 4) == 0)
+ memcpy (subdir, "cur/", 4);
+
+ strcpy (filename_new+(p-filename), ":2,");
+ strcpy (filename_new+(p-filename)+3, flags);
+
+ if (strcmp (filename, filename_new) != 0) {
+ ret = rename (filename, filename_new);
+ if (ret == -1) {
+ perror (talloc_asprintf (message, "rename of %s to %s failed", filename, filename_new));
+ exit (1);
+ }
+ return _notmuch_message_rename (message, filename_new);
+ /* _notmuch_message_sync is our caller. Do not call it here. */
+ }
+ return NOTMUCH_STATUS_SUCCESS;
+}
+
+static void
+tag_inbox_and_maildir_flags (notmuch_message_t *message, const char *filename)
+{
+ notmuch_message_add_tag (message, "inbox");
+ if (tag_from_maildir_flags(message, filename) != 0)
+ notmuch_message_add_tag (message, "unread");
+}
/* Examine 'path' recursively as follows:
*
@@ -171,6 +346,7 @@ add_files_recursive (notmuch_mailstore_t *mailstore,
struct stat st;
notmuch_bool_t is_maildir, new_directory;
_indexing_context_priv_t *priv = state->priv;
+ struct mailstore_priv *mailstore_priv = mailstore->priv;
notmuch_database_t *notmuch = mailstore->notmuch;
if (stat (path, &st)) {
@@ -352,11 +528,12 @@ add_files_recursive (notmuch_mailstore_t *mailstore,
/* success */
case NOTMUCH_STATUS_SUCCESS:
state->added_messages++;
- tag_inbox_and_unread (message);
+ mailstore_priv->tag_new (message, next);
break;
/* Non-fatal issues (go on to next file) */
case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
- /* Stay silent on this one. */
+ if (mailstore_priv->tag_renamed)
+ mailstore_priv->tag_renamed (message, next);
break;
case NOTMUCH_STATUS_FILE_NOT_EMAIL:
fprintf (stderr, "Note: Ignoring non-mail file: %s\n",
@@ -614,6 +791,10 @@ open_file(notmuch_mailstore_t *mailstore, const char *filename)
return fopen (abs_filename, "r");
}
+struct mailstore_priv files_priv = {
+ .tag_new = tag_inbox_and_unread,
+};
+
/* Original notmuch mail store */
struct _notmuch_mailstore mailstore_files = {
.type = "files",
@@ -621,4 +802,20 @@ struct _notmuch_mailstore mailstore_files = {
.index_new = index_new,
.sync_tags = NULL,
.open_file = open_file,
+ .priv = &files_priv,
+};
+
+struct mailstore_priv maildir_priv = {
+ .tag_new = tag_inbox_and_maildir_flags,
+ .tag_renamed = tag_from_maildir_flags,
+};
+
+/* Similar to mailstore_files, but does bi-directional synchronization between certain tags and maildir flags */
+struct _notmuch_mailstore mailstore_maildir = {
+ .type = "maildir",
+ .count_files = count_files,
+ .index_new = index_new,
+ .sync_tags = maildir_sync_tags,
+ .open_file = open_file,
+ .priv = &maildir_priv,
};
diff --git a/lib/mailstore.c b/lib/mailstore.c
index cdac9a6..7dc55ad 100644
--- a/lib/mailstore.c
+++ b/lib/mailstore.c
@@ -25,6 +25,7 @@
static notmuch_mailstore_t *available[] = {
&mailstore_files,
+ &mailstore_maildir,
};
notmuch_mailstore_t *
diff --git a/lib/message.cc b/lib/message.cc
index c32ee7d..51208ba 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -561,7 +561,8 @@ _notmuch_message_sync (notmuch_message_t *message)
if (message->notmuch->mode == NOTMUCH_DATABASE_MODE_READ_ONLY)
return;
- if (message->notmuch->mailstore->sync_tags) {
+ if (message->notmuch->mailstore->sync_tags &&
+ !notmuch_message_get_flag(message, NOTMUCH_MESSAGE_FLAG_TAGS_INVALID)) {
status = message->notmuch->mailstore->sync_tags (message->notmuch->mailstore,
message);
if (status != NOTMUCH_PRIVATE_STATUS_SUCCESS) {
@@ -683,6 +684,44 @@ _notmuch_message_remove_term (notmuch_message_t *message,
return NOTMUCH_PRIVATE_STATUS_SUCCESS;
}
+/* Change the message filename stored in the database.
+ *
+ * This change will not be reflected in the database until the next
+ * call to _notmuch_message_sync.
+ */
+notmuch_private_status_t
+_notmuch_message_rename (notmuch_message_t *message,
+ const char *new_filename)
+{
+ void *local = talloc_new (message);
+ char *direntry;
+ Xapian::PostingIterator i, end;
+ Xapian::Document document;
+ notmuch_private_status_t pstatus;
+ notmuch_status_t status;
+ const char *old_filename;
+
+ old_filename = notmuch_message_get_filename(message);
+ old_filename = talloc_reference(local, old_filename);
+ if (unlikely(!old_filename))
+ return NOTMUCH_PRIVATE_STATUS_OUT_OF_MEMORY;
+
+ status = _notmuch_message_add_filename (message, new_filename);
+ if (status)
+ return (notmuch_private_status_t)status;
+
+ status = _notmuch_database_filename_to_direntry (local, message->notmuch,
+ old_filename, &direntry);
+ if (status)
+ return (notmuch_private_status_t)status;
+
+ pstatus = _notmuch_message_remove_term (message, "file-direntry", direntry);
+
+ talloc_free (local);
+
+ return pstatus;
+}
+
notmuch_status_t
notmuch_message_add_tag (notmuch_message_t *message, const char *tag)
{
diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index d52d84d..63e75ec 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -245,6 +245,10 @@ notmuch_status_t
_notmuch_message_add_filename (notmuch_message_t *message,
const char *filename);
+notmuch_private_status_t
+_notmuch_message_rename (notmuch_message_t *message,
+ const char *new_filename);
+
void
_notmuch_message_ensure_thread_id (notmuch_message_t *message);
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 8fa7b31..c554f6d 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -760,6 +760,7 @@ notmuch_message_get_filename (notmuch_message_t *message);
/* Message flags */
typedef enum _notmuch_message_flag {
NOTMUCH_MESSAGE_FLAG_MATCH,
+ NOTMUCH_MESSAGE_FLAG_TAGS_INVALID,
} notmuch_message_flag_t;
/* Get a value of a flag for the email corresponding to 'message'. */
--
1.7.0.2
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [PATCH v2 4/4] Tests for maildir-based mailstore
2010-03-26 21:41 ` Michal Sojka
` (2 preceding siblings ...)
2010-03-26 21:42 ` [PATCH v2 3/4] Add maildir-based mailstore Michal Sojka
@ 2010-03-26 21:42 ` Michal Sojka
3 siblings, 0 replies; 25+ messages in thread
From: Michal Sojka @ 2010-03-26 21:42 UTC (permalink / raw)
To: notmuch
Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz>
---
test/t0006-maildir.sh | 205 +++++++++++++++++++++++++++++++++++++++++++++++++
test/test-lib.sh | 7 +-
2 files changed, 210 insertions(+), 2 deletions(-)
create mode 100755 test/t0006-maildir.sh
diff --git a/test/t0006-maildir.sh b/test/t0006-maildir.sh
new file mode 100755
index 0000000..ced1d05
--- /dev/null
+++ b/test/t0006-maildir.sh
@@ -0,0 +1,205 @@
+#!/bin/bash
+
+test_description="Test maildir mailstore"
+
+. ./test-lib.sh
+
+filter_output() {
+ grep -v -E -e "$NOTMUCH_IGNORED_OUTPUT_REGEXP" | sed -e "$NOTMUCH_THREAD_ID_SQUELCH"
+}
+
+filter_show() {
+ sed -e 's/, /,\n/g'|sed -e '/^"filename"/ s/:2,[A-Z]*//' -e '/^"tags"/d'
+}
+
+cat >> "$NOTMUCH_CONFIG" <<EOF
+[mailstore]
+type=maildir
+EOF
+
+test_expect_success "No new mail" '
+execute_expecting new "No new mail."
+'
+cat > expected <<EOF
+Added 1 new message to the database.
+EOF
+test_expect_success "Add a message, no flags" '
+generate_message [subject]="\"test message\"" [date]="\"Sat, 01 Jan 2000 12:00:00 -0000\"" [filename]="\"msg-001:2,\"" &&
+notmuch new | filter_output > actual &&
+test_cmp expected actual
+#emacs --eval "(gdb \"gdb --annotate=3 --args $(which notmuch) new\")"
+'
+cat > expected <<EOF
+thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test message (inbox unread)
+EOF
+test_expect_success 'Search for the message' '
+notmuch search tag:inbox and tag:unread | filter_output > actual &&
+test_cmp expected actual
+'
+cat > expected <<EOF
+No new mail. Detected 1 file rename.
+EOF
+test_expect_success 'Add seen flag' '
+mv "${gen_msg_filename}" "${gen_msg_filename}S" &&
+increment_mtime "$(dirname "${gen_msg_filename}")" &&
+notmuch new | filter_output > actual &&
+test_cmp expected actual
+'
+cat > expected <<EOF
+thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test message (inbox)
+EOF
+test_expect_success 'Check that tags were updated' '
+notmuch search tag:inbox and not tag:unread | filter_output > actual &&
+test_cmp expected actual
+'
+cat > expected <<EOF
+Added 1 new message to the database.
+EOF
+test_expect_success "Add a seen message" '
+generate_message [subject]="\"test message 2\"" [date]="\"Sat, 01 Jan 2000 12:00:00 -0000\"" [filename]="\"msg-002:2,S\"" &&
+notmuch new | filter_output > actual &&
+test_cmp expected actual
+'
+cat > expected <<EOF
+thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test message (inbox)
+thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test message 2 (inbox)
+EOF
+test_expect_success 'Check that the seen message is not tagged unread' '
+notmuch search tag:inbox and not tag:unread | filter_output > actual &&
+test_cmp expected actual
+'
+test_expect_success 'Tag the seen messages as replied' '
+notmuch tag +replied -inbox tag:inbox and not tag:unread
+'
+
+cat > expected <<EOF
+msg-001:2,RS
+msg-002:2,RS
+EOF
+test_expect_success 'Check that R flag was added' '
+ls -1 "${MAIL_DIR}" > actual &&
+test_cmp expected actual
+'
+echo -n '[[[{"id": "msg-001@notmuch-test-suite",
+"match": true,
+"filename": "/home/wsh/src/notmuch/test/trash directory.t0006-maildir/mail/msg-001",
+"headers": {"Subject": "test message",
+"From": "Notmuch Test Suite <test_suite@notmuchmail.org>",
+"To": "Notmuch Test Suite <test_suite@notmuchmail.org>",
+"Cc": "",
+"Bcc": "",
+"Date": "Sat,
+01 Jan 2000 12:00:00 -0000"},
+"body": [{"id": 1,
+"content-type": "text/plain",
+"content": "This is just a test message at /home/wsh/src/notmuch/test/trash directory.t0006-maildir/mail/msg-001:2,\n"}]},
+[]]]]' > show-expected
+
+test_expect_success 'Renamed message can be shown without running notmuch new' '
+notmuch show --format=json id:msg-001@notmuch-test-suite | filter_show > show-actual &&
+test_cmp show-expected show-actual
+'
+
+test_expect_success 'Test that we can reply to the renamed message' '
+notmuch reply id:msg-001@notmuch-test-suite
+'
+
+echo "No new mail." > expected
+test_expect_success 'No rename should be detected by notmuch new' '
+increment_mtime "$(dirname "${gen_msg_filename}")" &&
+notmuch new > actual &&
+test_cmp expected actual
+'
+test_expect_success "Add a message to new/ without info" '
+generate_message [subject]="\"test message 3\"" [date]="\"Sat, 01 Jan 2000 12:00:00 -0000\"" [dir]=new &&
+notmuch new | filter_output > actual &&
+test_cmp - actual <<EOF
+Added 1 new message to the database.
+EOF
+'
+test_expect_success "Check that the message has inbox and unread tags" '
+notmuch search tag:inbox and tag:unread | filter_output > actual &&
+test_cmp - actual <<EOF
+thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test message 3 (inbox unread)
+EOF
+'
+test_expect_success "Check that the message was not renamed" '
+ls "${MAIL_DIR}/new" > actual &&
+test_cmp - actual <<EOF
+msg-003
+EOF
+'
+test_expect_success 'Removing of unread tag should fail without cur/' '
+test_must_fail notmuch tag -unread tag:inbox and tag:unread
+'
+test_expect_success "Check that the tags were not changed" '
+notmuch search tag:inbox and tag:unread | filter_output > actual &&
+test_cmp - actual <<EOF
+thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test message 3 (inbox unread)
+EOF
+'
+
+# notmuch new is not necessary here, but we run it in order to check
+# for 'no rename' later (*).
+test_expect_success 'Create cur/ and let notmuch know about it' '
+mkdir "$MAIL_DIR/cur" &&
+notmuch new
+'
+test_expect_success 'Removing of unread tag should pass with cur/' '
+notmuch tag -unread tag:inbox and tag:unread
+'
+test_expect_success 'Check that the message was moved to cur/' '\
+ls "$MAIL_DIR/cur" > actual &&
+test_cmp - actual <<EOF
+msg-003:2,S
+EOF
+'
+test_expect_success 'No rename should be detected by notmuch new' '
+increment_mtime "$MAIL_DIR/cur" &&
+notmuch new > actual &&
+test_cmp - actual <<EOF
+No new mail.
+EOF
+'
+# (*) If notmuch new was not run we've got "Processed 1 file in almost
+# no time" here. The reason is that removing unread tag in a previous
+# test created directory document in the database but this document
+# was not linked as subdirectory of $MAIL_DIR. Therefore notmuch new
+# could not reach the cur/ directory and its files in it during
+# recurive traversal.
+test_expect_success 'Remove info from file name' '
+mv "$MAIL_DIR/cur/msg-003:2,S" "$MAIL_DIR/cur/msg-003" &&
+increment_mtime "$MAIL_DIR/cur" &&
+notmuch new | filter_output > actual
+test_cmp - actual <<EOF
+No new mail. Detected 1 file rename.
+EOF
+'
+test_expect_success "Check that removing info did not change tags" '
+notmuch search tag:inbox | filter_output > actual &&
+test_cmp - actual <<EOF
+thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test message 3 (inbox)
+EOF
+'
+test_expect_success "Add a message to fakenew/ without info" '
+generate_message [subject]="\"test message 4\"" [date]="\"Sat, 01 Jan 2000 12:00:00 -0000\"" [dir]=fakenew &&
+notmuch new | filter_output > actual &&
+test_cmp - actual <<EOF
+Added 1 new message to the database.
+EOF
+'
+test_expect_success "Check that the message has inbox and unread tags" '
+notmuch search tag:inbox and tag:unread | filter_output > actual &&
+test_cmp - actual <<EOF
+thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test message 4 (inbox unread)
+EOF
+'
+test_expect_success 'Removing of unread tag should leave the message in fakenew/' '
+notmuch tag -unread tag:inbox and tag:unread &&
+ls "$MAIL_DIR/fakenew" > actual &&
+test_cmp - actual <<EOF
+msg-004:2,S
+EOF
+'
+
+test_done
diff --git a/test/test-lib.sh b/test/test-lib.sh
index 5417fe7..917631b 100755
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -257,8 +257,11 @@ generate_message ()
local additional_headers
gen_msg_cnt=$((gen_msg_cnt + 1))
- gen_msg_name=msg-$(printf "%03d" $gen_msg_cnt)
- gen_msg_id="${gen_msg_name}@notmuch-test-suite"
+ if [ -z "${template[filename]}" ]; then
+ template[filename]="msg-$(printf "%03d" $gen_msg_cnt)"
+ fi
+ gen_msg_name=${template[filename]}
+ gen_msg_id="${gen_msg_name%:2,*}@notmuch-test-suite"
if [ -z "${template[dir]}" ]; then
gen_msg_filename="${MAIL_DIR}/$gen_msg_name"
--
1.7.0.2
^ permalink raw reply related [flat|nested] 25+ messages in thread
* Re: Mailstore abstraction & maildir synchronization
2010-03-24 5:06 ` Mailstore abstraction & maildir synchronization Stewart Smith
@ 2010-03-26 22:08 ` Michal Sojka
2010-03-27 20:42 ` Michal Sojka
0 siblings, 1 reply; 25+ messages in thread
From: Michal Sojka @ 2010-03-26 22:08 UTC (permalink / raw)
To: Stewart Smith, notmuch
On Wed, 24 Mar 2010, Stewart Smith wrote:
> On Thu, 18 Mar 2010 16:39:36 +0100, Michal Sojka <sojkam1@fel.cvut.cz> wrote:
> > - Only file-based storage is suported. Notmuch access the files
> > directly, and not via the mailstore interface.
>
> It'll be great when this is fixed... should be trivial to add a git
> backend then.
Yes, it seems to be quite trivial. I'll probably look at this tomorrow.
>
> (i have in no way been looking at tags in git though... doesn't really
> interest me and git aint a database)
My aim is only to store tags in git for the purpose of synchronization.
I'm not interested in searching by tags. The idea is that for every
message there will be an additional blob containging tags - one tag per
line.
>
> > - (maildir) Viewing/storing of attachments of unread messages doesn't
> > work. The reason is that when you view the message it its unread tag
> > is removed which leads to rename of the file, but Emacs still uses
> > the original name to access the attachment.
>
> What about migrating from a maildir that's turned into notmuch back to
> this maildir backend? What will be authoritive: maildir or notmuch
> database?
Maildir is authoritative, but only for tags corresponding to maildir
flags. Other tags are never touched by this code. See also the updated
commit message in "[PATCH v2 3/4] Add maildir-based mailstore".
If you want to sync your maildirs with the databse the following should
work (not tested):
notmuch dump > dump
cat <<EOF >> ~/.notmuch-config
[mailstore]
type=maildir
EOF
notmuch restore dump
--Michal
^ permalink raw reply [flat|nested] 25+ messages in thread
* Re: Mailstore abstraction & maildir synchronization
2010-03-26 22:08 ` Michal Sojka
@ 2010-03-27 20:42 ` Michal Sojka
2010-03-27 20:44 ` [PATCH v3 1/6] Mailstore abstraction interface Michal Sojka
` (6 more replies)
0 siblings, 7 replies; 25+ messages in thread
From: Michal Sojka @ 2010-03-27 20:42 UTC (permalink / raw)
To: Stewart Smith, notmuch
On Fri, 26 Mar 2010, Michal Sojka wrote:
> On Wed, 24 Mar 2010, Stewart Smith wrote:
> > On Thu, 18 Mar 2010 16:39:36 +0100, Michal Sojka <sojkam1@fel.cvut.cz> wrote:
> > > - Only file-based storage is suported. Notmuch access the files
> > > directly, and not via the mailstore interface.
> >
> > It'll be great when this is fixed... should be trivial to add a git
> > backend then.
>
> Yes, it seems to be quite trivial. I'll probably look at this tomorrow.
Here it is. It was not so trivial, because it was needed to change
absolute paths to relative ones at several places.
So the changes since v2 are:
- "Tests for maildir-based mailstore": removed absolute paths as found
on my computer.
- Added "Access messages through mail store interface"
- Added "Add 'cat' subcommand"
Michal Sojka (6):
Mailstore abstraction interface
Conversion to mailstore abstraction
Add maildir-based mailstore
Tests for maildir-based mailstore
Access messages through mail store interface
Add 'cat' subcommand
emacs/notmuch.el | 8 +-
lib/Makefile.local | 2 +
lib/database-private.h | 1 +
lib/database.cc | 36 ++-
lib/index.cc | 8 +-
lib/mailstore-files.c | 831 +++++++++++++++++++++++++++++++++++++++++++++++
lib/mailstore-private.h | 59 ++++
lib/mailstore.c | 78 +++++
lib/message-file.c | 8 +-
lib/message.cc | 85 +++++-
lib/notmuch-private.h | 10 +-
lib/notmuch.h | 98 ++++++-
lib/sha1.c | 6 +-
notmuch-client.h | 12 +-
notmuch-config.c | 34 ++
notmuch-count.c | 3 +-
notmuch-dump.c | 3 +-
notmuch-new.c | 646 ++++---------------------------------
notmuch-reply.c | 13 +-
notmuch-restore.c | 3 +-
notmuch-search-tags.c | 3 +-
notmuch-search.c | 3 +-
notmuch-show.c | 79 +++++-
notmuch-tag.c | 3 +-
notmuch.c | 4 +
show-message.c | 14 +-
test/t0006-maildir.sh | 208 ++++++++++++
test/test-lib.sh | 7 +-
28 files changed, 1611 insertions(+), 654 deletions(-)
create mode 100644 lib/mailstore-files.c
create mode 100644 lib/mailstore-private.h
create mode 100644 lib/mailstore.c
create mode 100755 test/t0006-maildir.sh
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v3 1/6] Mailstore abstraction interface
2010-03-27 20:42 ` Michal Sojka
@ 2010-03-27 20:44 ` Michal Sojka
2010-03-27 20:44 ` [PATCH v3 2/6] Conversion to mailstore abstraction Michal Sojka
` (5 subsequent siblings)
6 siblings, 0 replies; 25+ messages in thread
From: Michal Sojka @ 2010-03-27 20:44 UTC (permalink / raw)
To: notmuch
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 23517 bytes --]
The goal of mailstore abstraction is to allow notmuch to store tags
together with email messages. The abstract interface is needed because
people want to use different ways of storing their emails. This
patchseries implements two types of mailstore - plain files and
maildir. It is expected that additional git-based mailstore will
follow.
This patch contains only the interface changes. No functionality is
added, removed or changed.
A new configuration group [mailstore] is defined. Currently, there is
only one key 'type' whose value determines the used mailstore. The
default value of this option is the plain-file mailstore currently
implemented in notmuch.
Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz>
---
lib/Makefile.local | 2 +
lib/database-private.h | 1 +
lib/database.cc | 15 ++++++--
lib/mailstore-files.c | 43 +++++++++++++++++++++++++
lib/mailstore-private.h | 59 ++++++++++++++++++++++++++++++++++
lib/mailstore.c | 77 ++++++++++++++++++++++++++++++++++++++++++++
lib/message.cc | 13 +++++++
lib/notmuch.h | 81 ++++++++++++++++++++++++++++++++++++++++++++--
notmuch-client.h | 7 ++++
notmuch-config.c | 34 +++++++++++++++++++
notmuch-count.c | 3 +-
notmuch-dump.c | 3 +-
notmuch-new.c | 6 ++-
notmuch-reply.c | 3 +-
notmuch-restore.c | 3 +-
notmuch-search-tags.c | 3 +-
notmuch-search.c | 3 +-
notmuch-show.c | 3 +-
notmuch-tag.c | 3 +-
19 files changed, 344 insertions(+), 18 deletions(-)
create mode 100644 lib/mailstore-files.c
create mode 100644 lib/mailstore-private.h
create mode 100644 lib/mailstore.c
diff --git a/lib/Makefile.local b/lib/Makefile.local
index 495b27e..fc8883d 100644
--- a/lib/Makefile.local
+++ b/lib/Makefile.local
@@ -3,6 +3,8 @@ extra_cflags += -I$(dir)
libnotmuch_c_srcs = \
$(dir)/libsha1.c \
+ $(dir)/mailstore.c \
+ $(dir)/mailstore-files.c\
$(dir)/message-file.c \
$(dir)/messages.c \
$(dir)/sha1.c \
diff --git a/lib/database-private.h b/lib/database-private.h
index 41918d7..4499b1a 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -49,6 +49,7 @@ struct _notmuch_database {
Xapian::TermGenerator *term_gen;
Xapian::ValueRangeProcessor *value_range_processor;
+ notmuch_mailstore_t *mailstore;
};
/* Convert tags from Xapian internal format to notmuch format.
diff --git a/lib/database.cc b/lib/database.cc
index c91e97c..93c8d0f 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -19,6 +19,7 @@
*/
#include "database-private.h"
+#include "mailstore-private.h"
#include <iostream>
@@ -438,7 +439,7 @@ parse_references (void *ctx,
}
notmuch_database_t *
-notmuch_database_create (const char *path)
+notmuch_database_create (const char *path, notmuch_mailstore_t *mailstore)
{
notmuch_database_t *notmuch = NULL;
char *notmuch_path = NULL;
@@ -474,7 +475,8 @@ notmuch_database_create (const char *path)
}
notmuch = notmuch_database_open (path,
- NOTMUCH_DATABASE_MODE_READ_WRITE);
+ NOTMUCH_DATABASE_MODE_READ_WRITE,
+ mailstore);
notmuch_database_upgrade (notmuch, NULL, NULL);
DONE:
@@ -497,7 +499,8 @@ _notmuch_database_ensure_writable (notmuch_database_t *notmuch)
notmuch_database_t *
notmuch_database_open (const char *path,
- notmuch_database_mode_t mode)
+ notmuch_database_mode_t mode,
+ notmuch_mailstore_t *mailstore)
{
notmuch_database_t *notmuch = NULL;
char *notmuch_path = NULL, *xapian_path = NULL;
@@ -605,6 +608,9 @@ notmuch_database_open (const char *path,
prefix_t *prefix = &PROBABILISTIC_PREFIX[i];
notmuch->query_parser->add_prefix (prefix->name, prefix->prefix);
}
+
+ notmuch->mailstore = mailstore;
+ mailstore->notmuch = notmuch;
} catch (const Xapian::Error &error) {
fprintf (stderr, "A Xapian exception occurred opening database: %s\n",
error.get_msg().c_str());
@@ -1493,7 +1499,8 @@ notmuch_database_add_message (notmuch_database_t *notmuch,
DONE:
if (message) {
- if (ret == NOTMUCH_STATUS_SUCCESS && message_ret)
+ if ((ret == NOTMUCH_STATUS_SUCCESS ||
+ ret == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) && message_ret)
*message_ret = message;
else
notmuch_message_destroy (message);
diff --git a/lib/mailstore-files.c b/lib/mailstore-files.c
new file mode 100644
index 0000000..92d7f5d
--- /dev/null
+++ b/lib/mailstore-files.c
@@ -0,0 +1,43 @@
+/* mailstore-files.c - Original notmuch mail store - a collection of
+ * plain-text email messages (one message per file).
+ *
+ * Copyright © 2009 Carl Worth
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see http://www.gnu.org/licenses/ .
+ *
+ * Authors: Carl Worth <cworth@cworth.org>
+ * Michal Sojka <sojkam1@fel.cvut.cz>
+ */
+
+#include "notmuch.h"
+#include "mailstore-private.h"
+
+
+static FILE *
+open_file(notmuch_mailstore_t *mailstore, const char *filename)
+{
+ const char *db_path, *abs_filename;
+
+ db_path = notmuch_database_get_path(mailstore->notmuch);
+ abs_filename = talloc_asprintf(mailstore, "%s/%s", db_path, filename);
+ if (unlikely(abs_filename == NULL))
+ return NULL;
+ return fopen (abs_filename, "r");
+}
+
+/* Original notmuch mail store */
+struct _notmuch_mailstore mailstore_files = {
+ .type = "files",
+ .open_file = open_file,
+};
diff --git a/lib/mailstore-private.h b/lib/mailstore-private.h
new file mode 100644
index 0000000..f606e3f
--- /dev/null
+++ b/lib/mailstore-private.h
@@ -0,0 +1,59 @@
+/* mailstore-private.h - Mailstore abstraction
+ *
+ * Copyright © 2010 Michal Sojka
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see http://www.gnu.org/licenses/ .
+ *
+ * Author: Michal Sojka <sojkam1@fel.cvut.cz>
+ */
+
+#ifndef MAILSTORE_PRIVATE_H
+#define MAILSTORE_PRIVATE_H
+
+#include "notmuch-private.h"
+
+struct _notmuch_mailstore {
+ /* Constant fields */
+ const char *type; /* identification in .notmuch-config */
+ void *priv;
+
+ /* Count files in mailstore so that initial invocation of notmuch
+ * new can report remaining time */
+ void (*count_files)(notmuch_mailstore_t *mailstore,
+ const char *path, int *count,
+ volatile sig_atomic_t *interrupted);
+ /* Scan for new messages and add them to the database. */
+ notmuch_private_status_t (*index_new)(notmuch_mailstore_t *mailstore,
+ const char* path,
+ notmuch_indexing_context_t *ctx);
+ /* Store the tags assigned to the 'messages' to the
+ * 'mailstore'. */
+ notmuch_private_status_t (*sync_tags)(notmuch_mailstore_t *mailstore,
+ notmuch_message_t *message);
+ /* Return file handle from which the message can be read.
+ *
+ * Currently, this is not used and all message access is hardcoded
+ * to notmuch. We will need this operation for git-based and other
+ * non-filesystem mailstores. */
+ FILE * (*open_file)(notmuch_mailstore_t *mailstore,
+ const char *filename);
+
+ /* Run-time fields */
+ notmuch_database_t *notmuch;
+};
+
+extern struct _notmuch_mailstore mailstore_files;
+extern struct _notmuch_mailstore mailstore_maildir;
+
+#endif /* MAILSTORE_PRIVATE_H */
diff --git a/lib/mailstore.c b/lib/mailstore.c
new file mode 100644
index 0000000..cdac9a6
--- /dev/null
+++ b/lib/mailstore.c
@@ -0,0 +1,77 @@
+/* mailstore.c - Mailstore abstraction
+ *
+ * Copyright © 2010 Michal Sojka
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see http://www.gnu.org/licenses/ .
+ *
+ * Author: Michal Sojka <sojkam1@fel.cvut.cz>
+ */
+
+#include <notmuch.h>
+#include "mailstore-private.h"
+
+#define ARRAY_SIZE(arr) (sizeof (arr) / sizeof (arr[0]))
+
+static notmuch_mailstore_t *available[] = {
+ &mailstore_files,
+};
+
+notmuch_mailstore_t *
+notmuch_mailstore_get_by_type (const char *type)
+{
+ unsigned i;
+
+ if (type == NULL)
+ return available[0];
+
+ for (i = 0; i < ARRAY_SIZE(available); i++) {
+ if (strcasecmp (type, available[i]->type) == 0)
+ return available[i];
+ }
+ return NULL;
+}
+
+const char *
+notmuch_mailstore_get_type (notmuch_mailstore_t *mailstore)
+{
+ if (!mailstore)
+ return NULL;
+
+ return mailstore->type;
+}
+
+void
+notmuch_mailstore_count_files (notmuch_mailstore_t *mailstore,
+ const char *path, int *count,
+ volatile sig_atomic_t *interrupted)
+{
+ mailstore->count_files (mailstore, path, count, interrupted);
+}
+
+notmuch_status_t
+notmuch_mailstore_index_new (notmuch_mailstore_t *mailstore,
+ const char *path,
+ notmuch_indexing_context_t *ctx)
+{
+ notmuch_private_status_t status;
+ status = mailstore->index_new (mailstore, path, ctx);
+ return COERCE_STATUS (status, "Index new");
+}
+
+FILE *
+notmuch_mailstore_open_file (notmuch_mailstore_t *mailstore,
+ const char *filename)
+{
+ return mailstore->open_file (mailstore, filename);
+}
diff --git a/lib/message.cc b/lib/message.cc
index 721c9a6..c32ee7d 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -20,6 +20,7 @@
#include "notmuch-private.h"
#include "database-private.h"
+#include "mailstore-private.h"
#include <stdint.h>
@@ -555,10 +556,22 @@ void
_notmuch_message_sync (notmuch_message_t *message)
{
Xapian::WritableDatabase *db;
+ notmuch_private_status_t status;
if (message->notmuch->mode == NOTMUCH_DATABASE_MODE_READ_ONLY)
return;
+ if (message->notmuch->mailstore->sync_tags) {
+ status = message->notmuch->mailstore->sync_tags (message->notmuch->mailstore,
+ message);
+ if (status != NOTMUCH_PRIVATE_STATUS_SUCCESS) {
+ fprintf (stderr, "Error: Cannot sync tags to mailstore (%s)\n",
+ notmuch_status_to_string ((notmuch_status_t)status));
+ /* Exit to avoid unsynchronized mailstore. */
+ exit(1);
+ }
+ }
+
db = static_cast <Xapian::WritableDatabase *> (message->notmuch->xapian_db);
db->replace_document (message->doc_id, message->doc);
}
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 0d9cb0f..8fa7b31 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -32,6 +32,8 @@
NOTMUCH_BEGIN_DECLS
#include <time.h>
+#include <signal.h> /* For sig_atomic_t */
+#include <stdio.h> /* For FILE */
#ifndef FALSE
#define FALSE 0
@@ -119,6 +121,8 @@ typedef struct _notmuch_message notmuch_message_t;
typedef struct _notmuch_tags notmuch_tags_t;
typedef struct _notmuch_directory notmuch_directory_t;
typedef struct _notmuch_filenames notmuch_filenames_t;
+typedef struct _notmuch_mailstore notmuch_mailstore_t;
+
/* Create a new, empty notmuch database located at 'path'.
*
@@ -127,6 +131,8 @@ typedef struct _notmuch_filenames notmuch_filenames_t;
* create a new ".notmuch" directory within 'path' where notmuch will
* store its data.
*
+ * FIXME: Mailstore...
+ *
* After a successful call to notmuch_database_create, the returned
* database will be open so the caller should call
* notmuch_database_close when finished with it.
@@ -140,7 +146,7 @@ typedef struct _notmuch_filenames notmuch_filenames_t;
* an error message on stderr).
*/
notmuch_database_t *
-notmuch_database_create (const char *path);
+notmuch_database_create (const char *path, notmuch_mailstore_t *mailstore);
typedef enum {
NOTMUCH_DATABASE_MODE_READ_ONLY = 0,
@@ -152,6 +158,8 @@ typedef enum {
/* Open an existing notmuch database located at 'path'.
*
+ * The database will index mails stored in 'mailstore'.
+ *
* The database should have been created at some time in the past,
* (not necessarily by this process), by calling
* notmuch_database_create with 'path'. By default the database should be
@@ -169,7 +177,8 @@ typedef enum {
*/
notmuch_database_t *
notmuch_database_open (const char *path,
- notmuch_database_mode_t mode);
+ notmuch_database_mode_t mode,
+ notmuch_mailstore_t *mailstore);
/* Close the given notmuch database, freeing all associated
* resources. See notmuch_database_open. */
@@ -236,7 +245,8 @@ notmuch_database_get_directory (notmuch_database_t *database,
* notmuch database will reference the filename, and will not copy the
* entire contents of the file.
*
- * If 'message' is not NULL, then, on successful return '*message'
+ * If 'message' is not NULL, then, on successful return
+ * (NOTMUCH_STATUS_SUCCESS or NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) '*message'
* will be initialized to a message object that can be used for things
* such as adding tags to the just-added message. The user should call
* notmuch_message_destroy when done with the message. On any failure
@@ -495,7 +505,7 @@ notmuch_threads_destroy (notmuch_threads_t *threads);
*/
unsigned
notmuch_query_count_messages (notmuch_query_t *query);
-
+
/* Get the thread ID of 'thread'.
*
* The returned string belongs to 'thread' and as such, should not be
@@ -1104,6 +1114,69 @@ notmuch_filenames_move_to_next (notmuch_filenames_t *filenames);
void
notmuch_filenames_destroy (notmuch_filenames_t *filenames);
+/* Returns notmuch_mailstore_t object of a given 'type'.
+ *
+ * The 'type' is case insensitive. If 'type' is NULL, a default
+ * mailstore is returned.
+ *
+ * When no available mailstore matches 'type', NULL is returned.
+ */
+notmuch_mailstore_t *
+notmuch_mailstore_get_by_type (const char *type);
+
+/* Returns the type of a mailstore.
+ *
+ * The returned string is owned by mailstore so should not be modified
+ * nor freed by the caller.
+ */
+const char *
+notmuch_mailstore_get_type (notmuch_mailstore_t *mailstore);
+
+/* Count emails in mailstore.
+ *
+ * This function returns when interrupted is set to non-zero value.
+ */
+void
+notmuch_mailstore_count_files (notmuch_mailstore_t *mailstore,
+ const char *path, int *count,
+ volatile sig_atomic_t *interrupted);
+
+typedef struct notmuch_indexing_context {
+ int total_files;
+ int processed_files;
+ int added_messages;
+ int renamed_files;
+ int removed_files;
+
+ void *priv;
+
+ int verbose;
+ volatile sig_atomic_t interrupted;
+ volatile sig_atomic_t print_progress;
+ void (*print_progress_cb)(struct notmuch_indexing_context *ctx);
+ void (*print_verbose_cb)(struct notmuch_indexing_context *ctx, const char *filename);
+} notmuch_indexing_context_t;
+
+
+/* Index new mails in the mailstore.
+ *
+ * Mailstore is responsible for detecting new emails, adding them to
+ * the database and tagging them with tags stored in the mailstore.
+ */
+notmuch_status_t
+notmuch_mailstore_index_new (notmuch_mailstore_t *mailstore,
+ const char *path,
+ notmuch_indexing_context_t *ctx);
+
+/* Returns file handle to the file in the mailstore.
+ *
+ * TODO: This function is currently not used and access to mails is
+ * hardcoded in other parts of notmuch.
+ */
+FILE *
+notmuch_mailstore_open_file (notmuch_mailstore_t *mailstore,
+ const char *filename);
+
NOTMUCH_END_DECLS
#endif
diff --git a/notmuch-client.h b/notmuch-client.h
index c80b39c..4794b8e 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -171,6 +171,13 @@ notmuch_config_set_user_other_email (notmuch_config_t *config,
const char *other_email[],
size_t length);
+notmuch_mailstore_t *
+notmuch_config_get_mailstore(notmuch_config_t *config);
+
+void
+notmuch_config_set_mailstore(notmuch_config_t *config,
+ notmuch_mailstore_t *mailstore);
+
notmuch_bool_t
debugger_is_active (void);
diff --git a/notmuch-config.c b/notmuch-config.c
index 95430db..299423a 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -62,6 +62,7 @@ struct _notmuch_config {
char *user_primary_email;
char **user_other_email;
size_t user_other_email_length;
+ notmuch_mailstore_t *mailstore;
};
static int
@@ -199,6 +200,7 @@ notmuch_config_open (void *ctx,
config->user_primary_email = NULL;
config->user_other_email = NULL;
config->user_other_email_length = 0;
+ config->mailstore = NULL;
if (! g_key_file_load_from_file (config->key_file,
config->filename,
@@ -264,6 +266,10 @@ notmuch_config_open (void *ctx,
}
}
+ if (notmuch_config_get_mailstore (config) == NULL) {
+ notmuch_config_set_mailstore(config, notmuch_mailstore_get_by_type(NULL));
+ }
+
/* When we create a new configuration file here, we add some
* comments to help the user understand what can be done. */
if (is_new) {
@@ -452,3 +458,31 @@ notmuch_config_set_user_other_email (notmuch_config_t *config,
talloc_free (config->user_other_email);
config->user_other_email = NULL;
}
+
+notmuch_mailstore_t *
+notmuch_config_get_mailstore(notmuch_config_t *config)
+{
+ char *type;
+
+ if (config->mailstore == NULL) {
+ type = g_key_file_get_string (config->key_file,
+ "mailstore", "type", NULL);
+ if (type) {
+ config->mailstore = notmuch_mailstore_get_by_type(type);
+ free (type);
+ }
+ }
+
+ return config->mailstore;
+}
+
+void
+notmuch_config_set_mailstore(notmuch_config_t *config,
+ notmuch_mailstore_t *mailstore)
+{
+ g_key_file_set_string (config->key_file,
+ "mailstore", "type",
+ notmuch_mailstore_get_type(mailstore));
+
+ config->mailstore = NULL;
+}
diff --git a/notmuch-count.c b/notmuch-count.c
index 77aa433..631b8f8 100644
--- a/notmuch-count.c
+++ b/notmuch-count.c
@@ -81,7 +81,8 @@ notmuch_count_command (void *ctx, int argc, char *argv[])
return 1;
notmuch = notmuch_database_open (notmuch_config_get_database_path (config),
- NOTMUCH_DATABASE_MODE_READ_ONLY);
+ NOTMUCH_DATABASE_MODE_READ_ONLY,
+ notmuch_config_get_mailstore(config));
if (notmuch == NULL)
return 1;
diff --git a/notmuch-dump.c b/notmuch-dump.c
index 7e7bc17..e74dfcb 100644
--- a/notmuch-dump.c
+++ b/notmuch-dump.c
@@ -36,7 +36,8 @@ notmuch_dump_command (unused (void *ctx), int argc, char *argv[])
return 1;
notmuch = notmuch_database_open (notmuch_config_get_database_path (config),
- NOTMUCH_DATABASE_MODE_READ_ONLY);
+ NOTMUCH_DATABASE_MODE_READ_ONLY,
+ notmuch_config_get_mailstore (config));
if (notmuch == NULL)
return 1;
diff --git a/notmuch-new.c b/notmuch-new.c
index 44b50aa..2d0ba6c 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -749,11 +749,13 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
return 1;
printf ("Found %d total files (that's not much mail).\n", count);
- notmuch = notmuch_database_create (db_path);
+ notmuch = notmuch_database_create (db_path,
+ notmuch_config_get_mailstore (config));
add_files_state.total_files = count;
} else {
notmuch = notmuch_database_open (db_path,
- NOTMUCH_DATABASE_MODE_READ_WRITE);
+ NOTMUCH_DATABASE_MODE_READ_WRITE,
+ notmuch_config_get_mailstore (config));
if (notmuch == NULL)
return 1;
diff --git a/notmuch-reply.c b/notmuch-reply.c
index 6c15536..0d6b7cf 100644
--- a/notmuch-reply.c
+++ b/notmuch-reply.c
@@ -468,7 +468,8 @@ notmuch_reply_command (void *ctx, int argc, char *argv[])
}
notmuch = notmuch_database_open (notmuch_config_get_database_path (config),
- NOTMUCH_DATABASE_MODE_READ_ONLY);
+ NOTMUCH_DATABASE_MODE_READ_ONLY,
+ notmuch_config_get_mailstore (config));
if (notmuch == NULL)
return 1;
diff --git a/notmuch-restore.c b/notmuch-restore.c
index b0a4e1c..97d817b 100644
--- a/notmuch-restore.c
+++ b/notmuch-restore.c
@@ -37,7 +37,8 @@ notmuch_restore_command (unused (void *ctx), int argc, char *argv[])
return 1;
notmuch = notmuch_database_open (notmuch_config_get_database_path (config),
- NOTMUCH_DATABASE_MODE_READ_WRITE);
+ NOTMUCH_DATABASE_MODE_READ_WRITE,
+ notmuch_config_get_mailstore (config));
if (notmuch == NULL)
return 1;
diff --git a/notmuch-search-tags.c b/notmuch-search-tags.c
index 6f3cfcc..323f416 100644
--- a/notmuch-search-tags.c
+++ b/notmuch-search-tags.c
@@ -52,7 +52,8 @@ notmuch_search_tags_command (void *ctx, int argc, char *argv[])
}
db = notmuch_database_open (notmuch_config_get_database_path (config),
- NOTMUCH_DATABASE_MODE_READ_ONLY);
+ NOTMUCH_DATABASE_MODE_READ_ONLY,
+ notmuch_config_get_mailstore (config));
if (db == NULL) {
goto error;
}
diff --git a/notmuch-search.c b/notmuch-search.c
index 4e3514b..d57479d 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -245,7 +245,8 @@ notmuch_search_command (void *ctx, int argc, char *argv[])
return 1;
notmuch = notmuch_database_open (notmuch_config_get_database_path (config),
- NOTMUCH_DATABASE_MODE_READ_ONLY);
+ NOTMUCH_DATABASE_MODE_READ_ONLY,
+ notmuch_config_get_mailstore (config));
if (notmuch == NULL)
return 1;
diff --git a/notmuch-show.c b/notmuch-show.c
index ff1fecb..26581c1 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -458,7 +458,8 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
}
notmuch = notmuch_database_open (notmuch_config_get_database_path (config),
- NOTMUCH_DATABASE_MODE_READ_ONLY);
+ NOTMUCH_DATABASE_MODE_READ_ONLY,
+ notmuch_config_get_mailstore (config));
if (notmuch == NULL)
return 1;
diff --git a/notmuch-tag.c b/notmuch-tag.c
index 8b6f7dc..5e1a5c3 100644
--- a/notmuch-tag.c
+++ b/notmuch-tag.c
@@ -97,7 +97,8 @@ notmuch_tag_command (void *ctx, unused (int argc), unused (char *argv[]))
return 1;
notmuch = notmuch_database_open (notmuch_config_get_database_path (config),
- NOTMUCH_DATABASE_MODE_READ_WRITE);
+ NOTMUCH_DATABASE_MODE_READ_WRITE,
+ notmuch_config_get_mailstore (config));
if (notmuch == NULL)
return 1;
--
1.7.0.2
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [PATCH v3 2/6] Conversion to mailstore abstraction
2010-03-27 20:42 ` Michal Sojka
2010-03-27 20:44 ` [PATCH v3 1/6] Mailstore abstraction interface Michal Sojka
@ 2010-03-27 20:44 ` Michal Sojka
2010-03-27 20:44 ` [PATCH v3 3/6] Add maildir-based mailstore Michal Sojka
` (4 subsequent siblings)
6 siblings, 0 replies; 25+ messages in thread
From: Michal Sojka @ 2010-03-27 20:44 UTC (permalink / raw)
To: notmuch
The code for detection of new files in the mailstore and their
addition to the database is moved from notmuch-new.c to
lib/mailstore-files.c, where it is called by the abstract mailstore
interface.
The code was only changed to allow the progress reporting functions to
be implemented outside of notmuch library, as can be seen by
git diff HEAD^^ -C
Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz>
---
lib/mailstore-files.c | 581 ++++++++++++++++++++++++++++++++++++++++++++
notmuch-new.c | 648 +++++--------------------------------------------
2 files changed, 637 insertions(+), 592 deletions(-)
diff --git a/lib/mailstore-files.c b/lib/mailstore-files.c
index 92d7f5d..e4086ba 100644
--- a/lib/mailstore-files.c
+++ b/lib/mailstore-files.c
@@ -20,9 +20,587 @@
* Michal Sojka <sojkam1@fel.cvut.cz>
*/
+#define _GNU_SOURCE /* For asprintf() */
#include "notmuch.h"
#include "mailstore-private.h"
+#include <dirent.h>
+typedef struct _filename_node {
+ char *filename;
+ struct _filename_node *next;
+} _filename_node_t;
+
+typedef struct _filename_list {
+ _filename_node_t *head;
+ _filename_node_t **tail;
+} _filename_list_t;
+
+typedef struct _indexing_context_priv {
+ _filename_list_t *removed_files;
+ _filename_list_t *removed_directories;
+} _indexing_context_priv_t;
+
+static _filename_list_t *
+_filename_list_create (const void *ctx)
+{
+ _filename_list_t *list;
+
+ list = talloc (ctx, _filename_list_t);
+ if (list == NULL)
+ return NULL;
+
+ list->head = NULL;
+ list->tail = &list->head;
+
+ return list;
+}
+
+static void
+_filename_list_add (_filename_list_t *list,
+ const char *filename)
+{
+ _filename_node_t *node = talloc (list, _filename_node_t);
+
+ node->filename = talloc_strdup (list, filename);
+ node->next = NULL;
+
+ *(list->tail) = node;
+ list->tail = &node->next;
+}
+
+static void
+tag_inbox_and_unread (notmuch_message_t *message)
+{
+ notmuch_message_add_tag (message, "inbox");
+ notmuch_message_add_tag (message, "unread");
+}
+
+static int
+dirent_sort_inode (const struct dirent **a, const struct dirent **b)
+{
+ return ((*a)->d_ino < (*b)->d_ino) ? -1 : 1;
+}
+
+static int
+dirent_sort_strcmp_name (const struct dirent **a, const struct dirent **b)
+{
+ return strcmp ((*a)->d_name, (*b)->d_name);
+}
+
+/* Test if the directory looks like a Maildir directory.
+ *
+ * Search through the array of directory entries to see if we can find all
+ * three subdirectories typical for Maildir, that is "new", "cur", and "tmp".
+ *
+ * Return 1 if the directory looks like a Maildir and 0 otherwise.
+ */
+static int
+_entries_resemble_maildir (struct dirent **entries, int count)
+{
+ int i, found = 0;
+
+ for (i = 0; i < count; i++) {
+ if (entries[i]->d_type != DT_DIR && entries[i]->d_type != DT_UNKNOWN)
+ continue;
+
+ if (strcmp(entries[i]->d_name, "new") == 0 ||
+ strcmp(entries[i]->d_name, "cur") == 0 ||
+ strcmp(entries[i]->d_name, "tmp") == 0)
+ {
+ found++;
+ if (found == 3)
+ return 1;
+ }
+ }
+
+ return 0;
+}
+
+
+/* Examine 'path' recursively as follows:
+ *
+ * o Ask the filesystem for the mtime of 'path' (fs_mtime)
+ * o Ask the database for its timestamp of 'path' (db_mtime)
+ *
+ * o Ask the filesystem for files and directories within 'path'
+ * (via scandir and stored in fs_entries)
+ * o Ask the database for files and directories within 'path'
+ * (db_files and db_subdirs)
+ *
+ * o Pass 1: For each directory in fs_entries, recursively call into
+ * this same function.
+ *
+ * o Pass 2: If 'fs_mtime' > 'db_mtime', then walk fs_entries
+ * simultaneously with db_files and db_subdirs. Look for one of
+ * three interesting cases:
+ *
+ * 1. Regular file in fs_entries and not in db_files
+ * This is a new file to add_message into the database.
+ *
+ * 2. Filename in db_files not in fs_entries.
+ * This is a file that has been removed from the mail store.
+ *
+ * 3. Directory in db_subdirs not in fs_entries
+ * This is a directory that has been removed from the mail store.
+ *
+ * Note that the addition of a directory is not interesting here,
+ * since that will have been taken care of in pass 1. Also, we
+ * don't immediately act on file/directory removal since we must
+ * ensure that in the case of a rename that the new filename is
+ * added before the old filename is removed, (so that no
+ * information is lost from the database).
+ *
+ * o Tell the database to update its time of 'path' to 'fs_mtime'
+ */
+static notmuch_status_t
+add_files_recursive (notmuch_mailstore_t *mailstore,
+ const char *path,
+ notmuch_indexing_context_t *state)
+{
+ DIR *dir = NULL;
+ struct dirent *entry = NULL;
+ char *next = NULL;
+ time_t fs_mtime, db_mtime;
+ notmuch_status_t status, ret = NOTMUCH_STATUS_SUCCESS;
+ notmuch_message_t *message = NULL;
+ struct dirent **fs_entries = NULL;
+ int i, num_fs_entries;
+ notmuch_directory_t *directory;
+ notmuch_filenames_t *db_files = NULL;
+ notmuch_filenames_t *db_subdirs = NULL;
+ struct stat st;
+ notmuch_bool_t is_maildir, new_directory;
+ _indexing_context_priv_t *priv = state->priv;
+ notmuch_database_t *notmuch = mailstore->notmuch;
+
+ if (stat (path, &st)) {
+ fprintf (stderr, "Error reading directory %s: %s\n",
+ path, strerror (errno));
+ return NOTMUCH_STATUS_FILE_ERROR;
+ }
+
+ /* This is not an error since we may have recursed based on a
+ * symlink to a regular file, not a directory, and we don't know
+ * that until this stat. */
+ if (! S_ISDIR (st.st_mode))
+ return NOTMUCH_STATUS_SUCCESS;
+
+ fs_mtime = st.st_mtime;
+
+ directory = notmuch_database_get_directory (notmuch, path);
+ db_mtime = notmuch_directory_get_mtime (directory);
+
+ if (db_mtime == 0) {
+ new_directory = TRUE;
+ db_files = NULL;
+ db_subdirs = NULL;
+ } else {
+ new_directory = FALSE;
+ db_files = notmuch_directory_get_child_files (directory);
+ db_subdirs = notmuch_directory_get_child_directories (directory);
+ }
+
+ /* If the database knows about this directory, then we sort based
+ * on strcmp to match the database sorting. Otherwise, we can do
+ * inode-based sorting for faster filesystem operation. */
+ num_fs_entries = scandir (path, &fs_entries, 0,
+ new_directory ?
+ dirent_sort_inode : dirent_sort_strcmp_name);
+
+ if (num_fs_entries == -1) {
+ fprintf (stderr, "Error opening directory %s: %s\n",
+ path, strerror (errno));
+ ret = NOTMUCH_STATUS_FILE_ERROR;
+ goto DONE;
+ }
+
+ /* Pass 1: Recurse into all sub-directories. */
+ is_maildir = _entries_resemble_maildir (fs_entries, num_fs_entries);
+
+ for (i = 0; i < num_fs_entries; i++) {
+ if (state->interrupted)
+ break;
+
+ entry = fs_entries[i];
+
+ /* We only want to descend into directories.
+ * But symlinks can be to directories too, of course.
+ *
+ * And if the filesystem doesn't tell us the file type in the
+ * scandir results, then it might be a directory (and if not,
+ * then we'll stat and return immediately in the next level of
+ * recursion). */
+ if (entry->d_type != DT_DIR &&
+ entry->d_type != DT_LNK &&
+ entry->d_type != DT_UNKNOWN)
+ {
+ continue;
+ }
+
+ /* Ignore special directories to avoid infinite recursion.
+ * Also ignore the .notmuch directory and any "tmp" directory
+ * that appears within a maildir.
+ */
+ /* XXX: Eventually we'll want more sophistication to let the
+ * user specify files to be ignored. */
+ if (strcmp (entry->d_name, ".") == 0 ||
+ strcmp (entry->d_name, "..") == 0 ||
+ (is_maildir && strcmp (entry->d_name, "tmp") == 0) ||
+ strcmp (entry->d_name, ".notmuch") ==0)
+ {
+ continue;
+ }
+
+ next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
+ status = add_files_recursive (mailstore, next, state);
+ if (status && ret == NOTMUCH_STATUS_SUCCESS)
+ ret = status;
+ talloc_free (next);
+ next = NULL;
+ }
+
+ /* If this directory hasn't been modified since the last
+ * "notmuch new", then we can skip the second pass entirely. */
+ if (fs_mtime <= db_mtime)
+ goto DONE;
+
+ /* Pass 2: Scan for new files, removed files, and removed directories. */
+ for (i = 0; i < num_fs_entries; i++)
+ {
+ if (state->interrupted)
+ break;
+
+ entry = fs_entries[i];
+
+ /* Check if we've walked past any names in db_files or
+ * db_subdirs. If so, these have been deleted. */
+ while (notmuch_filenames_valid (db_files) &&
+ strcmp (notmuch_filenames_get (db_files), entry->d_name) < 0)
+ {
+ char *absolute = talloc_asprintf (priv->removed_files,
+ "%s/%s", path,
+ notmuch_filenames_get (db_files));
+
+ _filename_list_add (priv->removed_files, absolute);
+
+ notmuch_filenames_move_to_next (db_files);
+ }
+
+ while (notmuch_filenames_valid (db_subdirs) &&
+ strcmp (notmuch_filenames_get (db_subdirs), entry->d_name) <= 0)
+ {
+ const char *filename = notmuch_filenames_get (db_subdirs);
+
+ if (strcmp (filename, entry->d_name) < 0)
+ {
+ char *absolute = talloc_asprintf (priv->removed_directories,
+ "%s/%s", path, filename);
+
+ _filename_list_add (priv->removed_directories, absolute);
+ }
+
+ notmuch_filenames_move_to_next (db_subdirs);
+ }
+
+ /* If we're looking at a symlink, we only want to add it if it
+ * links to a regular file, (and not to a directory, say).
+ *
+ * Similarly, if the file is of unknown type (due to filesytem
+ * limitations), then we also need to look closer.
+ *
+ * In either case, a stat does the trick.
+ */
+ if (entry->d_type == DT_LNK || entry->d_type == DT_UNKNOWN) {
+ int err;
+
+ next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
+ err = stat (next, &st);
+ talloc_free (next);
+ next = NULL;
+
+ /* Don't emit an error for a link pointing nowhere, since
+ * the directory-traversal pass will have already done
+ * that. */
+ if (err)
+ continue;
+
+ if (! S_ISREG (st.st_mode))
+ continue;
+ } else if (entry->d_type != DT_REG) {
+ continue;
+ }
+
+ /* Don't add a file that we've added before. */
+ if (notmuch_filenames_valid (db_files) &&
+ strcmp (notmuch_filenames_get (db_files), entry->d_name) == 0)
+ {
+ notmuch_filenames_move_to_next (db_files);
+ continue;
+ }
+
+ /* We're now looking at a regular file that doesn't yet exist
+ * in the database, so add it. */
+ next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
+
+ state->processed_files++;
+
+ if (state->verbose && state->print_verbose_cb)
+ state->print_verbose_cb(state, next);
+
+ status = notmuch_database_add_message (notmuch, next, &message);
+ switch (status) {
+ /* success */
+ case NOTMUCH_STATUS_SUCCESS:
+ state->added_messages++;
+ tag_inbox_and_unread (message);
+ break;
+ /* Non-fatal issues (go on to next file) */
+ case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
+ /* Stay silent on this one. */
+ break;
+ case NOTMUCH_STATUS_FILE_NOT_EMAIL:
+ fprintf (stderr, "Note: Ignoring non-mail file: %s\n",
+ next);
+ break;
+ /* Fatal issues. Don't process anymore. */
+ case NOTMUCH_STATUS_READ_ONLY_DATABASE:
+ case NOTMUCH_STATUS_XAPIAN_EXCEPTION:
+ case NOTMUCH_STATUS_OUT_OF_MEMORY:
+ fprintf (stderr, "Error: %s. Halting processing.\n",
+ notmuch_status_to_string (status));
+ ret = status;
+ goto DONE;
+ default:
+ case NOTMUCH_STATUS_FILE_ERROR:
+ case NOTMUCH_STATUS_NULL_POINTER:
+ case NOTMUCH_STATUS_TAG_TOO_LONG:
+ case NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW:
+ case NOTMUCH_STATUS_LAST_STATUS:
+ INTERNAL_ERROR ("add_message returned unexpected value: %d", status);
+ goto DONE;
+ }
+
+ if (message) {
+ notmuch_message_destroy (message);
+ message = NULL;
+ }
+
+ if (state->print_progress &&
+ state->print_progress_cb) {
+ state->print_progress = 0;
+ state->print_progress_cb (state);
+ }
+
+ talloc_free (next);
+ next = NULL;
+ }
+
+ if (state->interrupted)
+ goto DONE;
+
+ /* Now that we've walked the whole filesystem list, anything left
+ * over in the database lists has been deleted. */
+ while (notmuch_filenames_valid (db_files))
+ {
+ char *absolute = talloc_asprintf (priv->removed_files,
+ "%s/%s", path,
+ notmuch_filenames_get (db_files));
+
+ _filename_list_add (priv->removed_files, absolute);
+
+ notmuch_filenames_move_to_next (db_files);
+ }
+
+ while (notmuch_filenames_valid (db_subdirs))
+ {
+ char *absolute = talloc_asprintf (priv->removed_directories,
+ "%s/%s", path,
+ notmuch_filenames_get (db_subdirs));
+
+ _filename_list_add (priv->removed_directories, absolute);
+
+ notmuch_filenames_move_to_next (db_subdirs);
+ }
+
+ if (! state->interrupted) {
+ status = notmuch_directory_set_mtime (directory, fs_mtime);
+ if (status && ret == NOTMUCH_STATUS_SUCCESS)
+ ret = status;
+ }
+
+DONE:
+ if (next)
+ talloc_free (next);
+ if (entry)
+ free (entry);
+ if (dir)
+ closedir (dir);
+ if (fs_entries)
+ free (fs_entries);
+ if (db_subdirs)
+ notmuch_filenames_destroy (db_subdirs);
+ if (db_files)
+ notmuch_filenames_destroy (db_files);
+ if (directory)
+ notmuch_directory_destroy (directory);
+
+ return ret;
+}
+
+/* XXX: This should be merged with the add_files function since it
+ * shares a lot of logic with it. */
+/* Recursively count all regular files in path and all sub-directories
+ * of path. The result is added to *count (which should be
+ * initialized to zero by the top-level caller before calling
+ * count_files). */
+static void
+count_files (notmuch_mailstore_t *mailstore,
+ const char *path, int *count,
+ volatile sig_atomic_t *interrupted)
+{
+ struct dirent *entry = NULL;
+ char *next;
+ struct stat st;
+ struct dirent **fs_entries = NULL;
+ int num_fs_entries = scandir (path, &fs_entries, 0, dirent_sort_inode);
+ int i = 0;
+
+ (void)mailstore;
+ if (num_fs_entries == -1) {
+ fprintf (stderr, "Warning: failed to open directory %s: %s\n",
+ path, strerror (errno));
+ goto DONE;
+ }
+
+ while (!*interrupted) {
+ if (i == num_fs_entries)
+ break;
+
+ entry = fs_entries[i++];
+
+ /* Ignore special directories to avoid infinite recursion.
+ * Also ignore the .notmuch directory.
+ */
+ /* XXX: Eventually we'll want more sophistication to let the
+ * user specify files to be ignored. */
+ if (strcmp (entry->d_name, ".") == 0 ||
+ strcmp (entry->d_name, "..") == 0 ||
+ strcmp (entry->d_name, ".notmuch") == 0)
+ {
+ continue;
+ }
+
+ if (asprintf (&next, "%s/%s", path, entry->d_name) == -1) {
+ next = NULL;
+ fprintf (stderr, "Error descending from %s to %s: Out of memory\n",
+ path, entry->d_name);
+ continue;
+ }
+
+ stat (next, &st);
+
+ if (S_ISREG (st.st_mode)) {
+ *count = *count + 1;
+ if (*count % 1000 == 0) {
+ printf ("Found %d files so far.\r", *count);
+ fflush (stdout);
+ }
+ } else if (S_ISDIR (st.st_mode)) {
+ count_files (mailstore, next, count, interrupted);
+ }
+
+ free (next);
+ }
+
+DONE:
+ if (entry)
+ free (entry);
+ if (fs_entries)
+ free (fs_entries);
+}
+
+/* Recursively remove all filenames from the database referring to
+ * 'path' (or to any of its children). */
+static void
+_remove_directory (void *ctx,
+ notmuch_database_t *notmuch,
+ const char *path,
+ int *renamed_files,
+ int *removed_files)
+{
+ notmuch_directory_t *directory;
+ notmuch_filenames_t *files, *subdirs;
+ notmuch_status_t status;
+ char *absolute;
+
+ directory = notmuch_database_get_directory (notmuch, path);
+
+ for (files = notmuch_directory_get_child_files (directory);
+ notmuch_filenames_valid (files);
+ notmuch_filenames_move_to_next (files))
+ {
+ absolute = talloc_asprintf (ctx, "%s/%s", path,
+ notmuch_filenames_get (files));
+ status = notmuch_database_remove_message (notmuch, absolute);
+ if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID)
+ *renamed_files = *renamed_files + 1;
+ else
+ *removed_files = *removed_files + 1;
+ talloc_free (absolute);
+ }
+
+ for (subdirs = notmuch_directory_get_child_directories (directory);
+ notmuch_filenames_valid (subdirs);
+ notmuch_filenames_move_to_next (subdirs))
+ {
+ absolute = talloc_asprintf (ctx, "%s/%s", path,
+ notmuch_filenames_get (subdirs));
+ _remove_directory (ctx, notmuch, absolute, renamed_files, removed_files);
+ talloc_free (absolute);
+ }
+
+ notmuch_directory_destroy (directory);
+}
+
+static notmuch_private_status_t
+index_new(notmuch_mailstore_t *mailstore, const char* path,
+ notmuch_indexing_context_t *indexing_ctx)
+{
+ _indexing_context_priv_t *priv;
+ _filename_node_t *f;
+ notmuch_status_t status, ret;
+ notmuch_database_t *notmuch = mailstore->notmuch;
+
+ priv = talloc(NULL, _indexing_context_priv_t);
+ indexing_ctx->priv = priv;
+ if (priv == NULL)
+ return NOTMUCH_STATUS_OUT_OF_MEMORY;
+
+ priv->removed_files = _filename_list_create (priv);
+ priv->removed_directories = _filename_list_create (priv);
+
+ ret = add_files_recursive(mailstore, path, indexing_ctx);
+
+ indexing_ctx->removed_files = 0;
+ indexing_ctx->renamed_files = 0;
+ for (f = priv->removed_files->head; f; f = f->next) {
+ status = notmuch_database_remove_message (notmuch, f->filename);
+ if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID)
+ indexing_ctx->renamed_files++;
+ else
+ indexing_ctx->removed_files++;
+ }
+
+ for (f = priv->removed_directories->head; f; f = f->next) {
+ _remove_directory (priv, notmuch, f->filename,
+ &indexing_ctx->renamed_files,
+ &indexing_ctx->removed_files);
+ }
+
+ talloc_free(priv);
+
+ return ret;
+}
static FILE *
open_file(notmuch_mailstore_t *mailstore, const char *filename)
@@ -39,5 +617,8 @@ open_file(notmuch_mailstore_t *mailstore, const char *filename)
/* Original notmuch mail store */
struct _notmuch_mailstore mailstore_files = {
.type = "files",
+ .count_files = count_files,
+ .index_new = index_new,
+ .sync_tags = NULL,
.open_file = open_file,
};
diff --git a/notmuch-new.c b/notmuch-new.c
index 2d0ba6c..1ff5840 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -22,35 +22,12 @@
#include <unistd.h>
-typedef struct _filename_node {
- char *filename;
- struct _filename_node *next;
-} _filename_node_t;
-
-typedef struct _filename_list {
- _filename_node_t *head;
- _filename_node_t **tail;
-} _filename_list_t;
-
-typedef struct {
- int output_is_a_tty;
- int verbose;
-
- int total_files;
- int processed_files;
- int added_messages;
- struct timeval tv_start;
-
- _filename_list_t *removed_files;
- _filename_list_t *removed_directories;
-} add_files_state_t;
-
-static volatile sig_atomic_t do_add_files_print_progress = 0;
+notmuch_indexing_context_t *indexing_ctx;
static void
handle_sigalrm (unused (int signal))
{
- do_add_files_print_progress = 1;
+ indexing_ctx->print_progress = 1;
}
static volatile sig_atomic_t interrupted;
@@ -62,53 +39,24 @@ handle_sigint (unused (int sig))
static char msg[] = "Stopping... \n";
ignored = write(2, msg, sizeof(msg)-1);
- interrupted = 1;
+ indexing_ctx->interrupted = 1;
}
-static _filename_list_t *
-_filename_list_create (const void *ctx)
-{
- _filename_list_t *list;
-
- list = talloc (ctx, _filename_list_t);
- if (list == NULL)
- return NULL;
-
- list->head = NULL;
- list->tail = &list->head;
-
- return list;
-}
-
-static void
-_filename_list_add (_filename_list_t *list,
- const char *filename)
-{
- _filename_node_t *node = talloc (list, _filename_node_t);
-
- node->filename = talloc_strdup (list, filename);
- node->next = NULL;
-
- *(list->tail) = node;
- list->tail = &node->next;
-}
-
-static void
-tag_inbox_and_unread (notmuch_message_t *message)
-{
- notmuch_message_add_tag (message, "inbox");
- notmuch_message_add_tag (message, "unread");
-}
+struct priv_ctx {
+ struct timeval tv_start;
+ int output_is_a_tty;
+};
static void
-add_files_print_progress (add_files_state_t *state)
+add_files_print_progress (notmuch_indexing_context_t *state)
{
+ struct priv_ctx *state2 = state->priv;
struct timeval tv_now;
double elapsed_overall, rate_overall;
gettimeofday (&tv_now, NULL);
- elapsed_overall = notmuch_time_elapsed (state->tv_start, tv_now);
+ elapsed_overall = notmuch_time_elapsed (state2->tv_start, tv_now);
rate_overall = (state->processed_files) / elapsed_overall;
printf ("Processed %d", state->processed_files);
@@ -128,396 +76,42 @@ add_files_print_progress (add_files_state_t *state)
fflush (stdout);
}
-static int
-dirent_sort_inode (const struct dirent **a, const struct dirent **b)
-{
- return ((*a)->d_ino < (*b)->d_ino) ? -1 : 1;
-}
-
-static int
-dirent_sort_strcmp_name (const struct dirent **a, const struct dirent **b)
-{
- return strcmp ((*a)->d_name, (*b)->d_name);
-}
-
-/* Test if the directory looks like a Maildir directory.
- *
- * Search through the array of directory entries to see if we can find all
- * three subdirectories typical for Maildir, that is "new", "cur", and "tmp".
- *
- * Return 1 if the directory looks like a Maildir and 0 otherwise.
- */
-static int
-_entries_resemble_maildir (struct dirent **entries, int count)
-{
- int i, found = 0;
-
- for (i = 0; i < count; i++) {
- if (entries[i]->d_type != DT_DIR && entries[i]->d_type != DT_UNKNOWN)
- continue;
-
- if (strcmp(entries[i]->d_name, "new") == 0 ||
- strcmp(entries[i]->d_name, "cur") == 0 ||
- strcmp(entries[i]->d_name, "tmp") == 0)
- {
- found++;
- if (found == 3)
- return 1;
- }
- }
-
- return 0;
-}
-
-/* Examine 'path' recursively as follows:
- *
- * o Ask the filesystem for the mtime of 'path' (fs_mtime)
- * o Ask the database for its timestamp of 'path' (db_mtime)
- *
- * o Ask the filesystem for files and directories within 'path'
- * (via scandir and stored in fs_entries)
- * o Ask the database for files and directories within 'path'
- * (db_files and db_subdirs)
- *
- * o Pass 1: For each directory in fs_entries, recursively call into
- * this same function.
- *
- * o Pass 2: If 'fs_mtime' > 'db_mtime', then walk fs_entries
- * simultaneously with db_files and db_subdirs. Look for one of
- * three interesting cases:
- *
- * 1. Regular file in fs_entries and not in db_files
- * This is a new file to add_message into the database.
- *
- * 2. Filename in db_files not in fs_entries.
- * This is a file that has been removed from the mail store.
- *
- * 3. Directory in db_subdirs not in fs_entries
- * This is a directory that has been removed from the mail store.
- *
- * Note that the addition of a directory is not interesting here,
- * since that will have been taken care of in pass 1. Also, we
- * don't immediately act on file/directory removal since we must
- * ensure that in the case of a rename that the new filename is
- * added before the old filename is removed, (so that no
- * information is lost from the database).
- *
- * o Tell the database to update its time of 'path' to 'fs_mtime'
- */
-static notmuch_status_t
-add_files_recursive (notmuch_database_t *notmuch,
- const char *path,
- add_files_state_t *state)
+static void
+add_files_print_verbose (notmuch_indexing_context_t *state, const char *filename)
{
- DIR *dir = NULL;
- struct dirent *entry = NULL;
- char *next = NULL;
- time_t fs_mtime, db_mtime;
- notmuch_status_t status, ret = NOTMUCH_STATUS_SUCCESS;
- notmuch_message_t *message = NULL;
- struct dirent **fs_entries = NULL;
- int i, num_fs_entries;
- notmuch_directory_t *directory;
- notmuch_filenames_t *db_files = NULL;
- notmuch_filenames_t *db_subdirs = NULL;
- struct stat st;
- notmuch_bool_t is_maildir, new_directory;
-
- if (stat (path, &st)) {
- fprintf (stderr, "Error reading directory %s: %s\n",
- path, strerror (errno));
- return NOTMUCH_STATUS_FILE_ERROR;
- }
-
- /* This is not an error since we may have recursed based on a
- * symlink to a regular file, not a directory, and we don't know
- * that until this stat. */
- if (! S_ISDIR (st.st_mode))
- return NOTMUCH_STATUS_SUCCESS;
-
- fs_mtime = st.st_mtime;
-
- directory = notmuch_database_get_directory (notmuch, path);
- db_mtime = notmuch_directory_get_mtime (directory);
-
- if (db_mtime == 0) {
- new_directory = TRUE;
- db_files = NULL;
- db_subdirs = NULL;
- } else {
- new_directory = FALSE;
- db_files = notmuch_directory_get_child_files (directory);
- db_subdirs = notmuch_directory_get_child_directories (directory);
- }
-
- /* If the database knows about this directory, then we sort based
- * on strcmp to match the database sorting. Otherwise, we can do
- * inode-based sorting for faster filesystem operation. */
- num_fs_entries = scandir (path, &fs_entries, 0,
- new_directory ?
- dirent_sort_inode : dirent_sort_strcmp_name);
-
- if (num_fs_entries == -1) {
- fprintf (stderr, "Error opening directory %s: %s\n",
- path, strerror (errno));
- ret = NOTMUCH_STATUS_FILE_ERROR;
- goto DONE;
- }
-
- /* Pass 1: Recurse into all sub-directories. */
- is_maildir = _entries_resemble_maildir (fs_entries, num_fs_entries);
-
- for (i = 0; i < num_fs_entries; i++) {
- if (interrupted)
- break;
-
- entry = fs_entries[i];
-
- /* We only want to descend into directories.
- * But symlinks can be to directories too, of course.
- *
- * And if the filesystem doesn't tell us the file type in the
- * scandir results, then it might be a directory (and if not,
- * then we'll stat and return immediately in the next level of
- * recursion). */
- if (entry->d_type != DT_DIR &&
- entry->d_type != DT_LNK &&
- entry->d_type != DT_UNKNOWN)
- {
- continue;
- }
-
- /* Ignore special directories to avoid infinite recursion.
- * Also ignore the .notmuch directory and any "tmp" directory
- * that appears within a maildir.
- */
- /* XXX: Eventually we'll want more sophistication to let the
- * user specify files to be ignored. */
- if (strcmp (entry->d_name, ".") == 0 ||
- strcmp (entry->d_name, "..") == 0 ||
- (is_maildir && strcmp (entry->d_name, "tmp") == 0) ||
- strcmp (entry->d_name, ".notmuch") ==0)
- {
- continue;
- }
-
- next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
- status = add_files_recursive (notmuch, next, state);
- if (status && ret == NOTMUCH_STATUS_SUCCESS)
- ret = status;
- talloc_free (next);
- next = NULL;
- }
-
- /* If this directory hasn't been modified since the last
- * "notmuch new", then we can skip the second pass entirely. */
- if (fs_mtime <= db_mtime)
- goto DONE;
-
- /* Pass 2: Scan for new files, removed files, and removed directories. */
- for (i = 0; i < num_fs_entries; i++)
- {
- if (interrupted)
- break;
+ struct priv_ctx *state2 = state->priv;
+ if (state2->output_is_a_tty)
+ printf("\r\033[K");
- entry = fs_entries[i];
+ printf ("%i/%i: %s",
+ state->processed_files,
+ state->total_files,
+ filename);
- /* Check if we've walked past any names in db_files or
- * db_subdirs. If so, these have been deleted. */
- while (notmuch_filenames_valid (db_files) &&
- strcmp (notmuch_filenames_get (db_files), entry->d_name) < 0)
- {
- char *absolute = talloc_asprintf (state->removed_files,
- "%s/%s", path,
- notmuch_filenames_get (db_files));
-
- _filename_list_add (state->removed_files, absolute);
-
- notmuch_filenames_move_to_next (db_files);
- }
-
- while (notmuch_filenames_valid (db_subdirs) &&
- strcmp (notmuch_filenames_get (db_subdirs), entry->d_name) <= 0)
- {
- const char *filename = notmuch_filenames_get (db_subdirs);
-
- if (strcmp (filename, entry->d_name) < 0)
- {
- char *absolute = talloc_asprintf (state->removed_directories,
- "%s/%s", path, filename);
-
- _filename_list_add (state->removed_directories, absolute);
- }
-
- notmuch_filenames_move_to_next (db_subdirs);
- }
-
- /* If we're looking at a symlink, we only want to add it if it
- * links to a regular file, (and not to a directory, say).
- *
- * Similarly, if the file is of unknown type (due to filesytem
- * limitations), then we also need to look closer.
- *
- * In either case, a stat does the trick.
- */
- if (entry->d_type == DT_LNK || entry->d_type == DT_UNKNOWN) {
- int err;
-
- next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
- err = stat (next, &st);
- talloc_free (next);
- next = NULL;
-
- /* Don't emit an error for a link pointing nowhere, since
- * the directory-traversal pass will have already done
- * that. */
- if (err)
- continue;
-
- if (! S_ISREG (st.st_mode))
- continue;
- } else if (entry->d_type != DT_REG) {
- continue;
- }
-
- /* Don't add a file that we've added before. */
- if (notmuch_filenames_valid (db_files) &&
- strcmp (notmuch_filenames_get (db_files), entry->d_name) == 0)
- {
- notmuch_filenames_move_to_next (db_files);
- continue;
- }
-
- /* We're now looking at a regular file that doesn't yet exist
- * in the database, so add it. */
- next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
-
- state->processed_files++;
-
- if (state->verbose) {
- if (state->output_is_a_tty)
- printf("\r\033[K");
-
- printf ("%i/%i: %s",
- state->processed_files,
- state->total_files,
- next);
-
- putchar((state->output_is_a_tty) ? '\r' : '\n');
- fflush (stdout);
- }
-
- status = notmuch_database_add_message (notmuch, next, &message);
- switch (status) {
- /* success */
- case NOTMUCH_STATUS_SUCCESS:
- state->added_messages++;
- tag_inbox_and_unread (message);
- break;
- /* Non-fatal issues (go on to next file) */
- case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
- /* Stay silent on this one. */
- break;
- case NOTMUCH_STATUS_FILE_NOT_EMAIL:
- fprintf (stderr, "Note: Ignoring non-mail file: %s\n",
- next);
- break;
- /* Fatal issues. Don't process anymore. */
- case NOTMUCH_STATUS_READ_ONLY_DATABASE:
- case NOTMUCH_STATUS_XAPIAN_EXCEPTION:
- case NOTMUCH_STATUS_OUT_OF_MEMORY:
- fprintf (stderr, "Error: %s. Halting processing.\n",
- notmuch_status_to_string (status));
- ret = status;
- goto DONE;
- default:
- case NOTMUCH_STATUS_FILE_ERROR:
- case NOTMUCH_STATUS_NULL_POINTER:
- case NOTMUCH_STATUS_TAG_TOO_LONG:
- case NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW:
- case NOTMUCH_STATUS_LAST_STATUS:
- INTERNAL_ERROR ("add_message returned unexpected value: %d", status);
- goto DONE;
- }
-
- if (message) {
- notmuch_message_destroy (message);
- message = NULL;
- }
-
- if (do_add_files_print_progress) {
- do_add_files_print_progress = 0;
- add_files_print_progress (state);
- }
-
- talloc_free (next);
- next = NULL;
- }
-
- /* Now that we've walked the whole filesystem list, anything left
- * over in the database lists has been deleted. */
- while (notmuch_filenames_valid (db_files))
- {
- char *absolute = talloc_asprintf (state->removed_files,
- "%s/%s", path,
- notmuch_filenames_get (db_files));
-
- _filename_list_add (state->removed_files, absolute);
-
- notmuch_filenames_move_to_next (db_files);
- }
-
- while (notmuch_filenames_valid (db_subdirs))
- {
- char *absolute = talloc_asprintf (state->removed_directories,
- "%s/%s", path,
- notmuch_filenames_get (db_subdirs));
-
- _filename_list_add (state->removed_directories, absolute);
-
- notmuch_filenames_move_to_next (db_subdirs);
- }
-
- if (! interrupted) {
- status = notmuch_directory_set_mtime (directory, fs_mtime);
- if (status && ret == NOTMUCH_STATUS_SUCCESS)
- ret = status;
- }
-
- DONE:
- if (next)
- talloc_free (next);
- if (entry)
- free (entry);
- if (dir)
- closedir (dir);
- if (fs_entries)
- free (fs_entries);
- if (db_subdirs)
- notmuch_filenames_destroy (db_subdirs);
- if (db_files)
- notmuch_filenames_destroy (db_files);
- if (directory)
- notmuch_directory_destroy (directory);
-
- return ret;
+ putchar((state2->output_is_a_tty) ? '\r' : '\n');
+ fflush (stdout);
}
/* This is the top-level entry point for add_files. It does a couple
* of error checks, sets up the progress-printing timer and then calls
* into the recursive function. */
static notmuch_status_t
-add_files (notmuch_database_t *notmuch,
+add_files (notmuch_mailstore_t *mailstore,
const char *path,
- add_files_state_t *state)
+ notmuch_indexing_context_t *state) /* FIXME: rename */
{
+ struct priv_ctx *state2 = state->priv;
notmuch_status_t status;
struct sigaction action;
struct itimerval timerval;
notmuch_bool_t timer_is_active = FALSE;
struct stat st;
- if (state->output_is_a_tty && ! debugger_is_active () && ! state->verbose) {
+ state->print_progress = 0;
+ state->print_progress_cb = add_files_print_progress;
+ state->print_verbose_cb = add_files_print_verbose;
+
+ if (state2->output_is_a_tty && ! debugger_is_active () && ! state->verbose) {
/* Setup our handler for SIGALRM */
memset (&action, 0, sizeof (struct sigaction));
action.sa_handler = handle_sigalrm;
@@ -546,7 +140,7 @@ add_files (notmuch_database_t *notmuch,
return NOTMUCH_STATUS_FILE_ERROR;
}
- status = add_files_recursive (notmuch, path, state);
+ status = notmuch_mailstore_index_new (mailstore, path, state);
if (timer_is_active) {
/* Now stop the timer. */
@@ -564,80 +158,12 @@ add_files (notmuch_database_t *notmuch,
return status;
}
-/* XXX: This should be merged with the add_files function since it
- * shares a lot of logic with it. */
-/* Recursively count all regular files in path and all sub-directories
- * of path. The result is added to *count (which should be
- * initialized to zero by the top-level caller before calling
- * count_files). */
-static void
-count_files (const char *path, int *count)
-{
- struct dirent *entry = NULL;
- char *next;
- struct stat st;
- struct dirent **fs_entries = NULL;
- int num_fs_entries = scandir (path, &fs_entries, 0, dirent_sort_inode);
- int i = 0;
-
- if (num_fs_entries == -1) {
- fprintf (stderr, "Warning: failed to open directory %s: %s\n",
- path, strerror (errno));
- goto DONE;
- }
-
- while (!interrupted) {
- if (i == num_fs_entries)
- break;
-
- entry = fs_entries[i++];
-
- /* Ignore special directories to avoid infinite recursion.
- * Also ignore the .notmuch directory.
- */
- /* XXX: Eventually we'll want more sophistication to let the
- * user specify files to be ignored. */
- if (strcmp (entry->d_name, ".") == 0 ||
- strcmp (entry->d_name, "..") == 0 ||
- strcmp (entry->d_name, ".notmuch") == 0)
- {
- continue;
- }
-
- if (asprintf (&next, "%s/%s", path, entry->d_name) == -1) {
- next = NULL;
- fprintf (stderr, "Error descending from %s to %s: Out of memory\n",
- path, entry->d_name);
- continue;
- }
-
- stat (next, &st);
-
- if (S_ISREG (st.st_mode)) {
- *count = *count + 1;
- if (*count % 1000 == 0) {
- printf ("Found %d files so far.\r", *count);
- fflush (stdout);
- }
- } else if (S_ISDIR (st.st_mode)) {
- count_files (next, count);
- }
-
- free (next);
- }
-
- DONE:
- if (entry)
- free (entry);
- if (fs_entries)
- free (fs_entries);
-}
-
static void
upgrade_print_progress (void *closure,
double progress)
{
- add_files_state_t *state = closure;
+ notmuch_indexing_context_t *state = closure;
+ struct priv_ctx *state2 = state->priv;
printf ("Upgrading database: %.2f%% complete", progress * 100.0);
@@ -647,7 +173,7 @@ upgrade_print_progress (void *closure,
gettimeofday (&tv_now, NULL);
- elapsed = notmuch_time_elapsed (state->tv_start, tv_now);
+ elapsed = notmuch_time_elapsed (state2->tv_start, tv_now);
time_remaining = (elapsed / progress) * (1.0 - progress);
printf (" (");
notmuch_time_print_formatted_seconds (time_remaining);
@@ -659,55 +185,13 @@ upgrade_print_progress (void *closure,
fflush (stdout);
}
-/* Recursively remove all filenames from the database referring to
- * 'path' (or to any of its children). */
-static void
-_remove_directory (void *ctx,
- notmuch_database_t *notmuch,
- const char *path,
- int *renamed_files,
- int *removed_files)
-{
- notmuch_directory_t *directory;
- notmuch_filenames_t *files, *subdirs;
- notmuch_status_t status;
- char *absolute;
-
- directory = notmuch_database_get_directory (notmuch, path);
-
- for (files = notmuch_directory_get_child_files (directory);
- notmuch_filenames_valid (files);
- notmuch_filenames_move_to_next (files))
- {
- absolute = talloc_asprintf (ctx, "%s/%s", path,
- notmuch_filenames_get (files));
- status = notmuch_database_remove_message (notmuch, absolute);
- if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID)
- *renamed_files = *renamed_files + 1;
- else
- *removed_files = *removed_files + 1;
- talloc_free (absolute);
- }
-
- for (subdirs = notmuch_directory_get_child_directories (directory);
- notmuch_filenames_valid (subdirs);
- notmuch_filenames_move_to_next (subdirs))
- {
- absolute = talloc_asprintf (ctx, "%s/%s", path,
- notmuch_filenames_get (subdirs));
- _remove_directory (ctx, notmuch, absolute, renamed_files, removed_files);
- talloc_free (absolute);
- }
-
- notmuch_directory_destroy (directory);
-}
-
int
notmuch_new_command (void *ctx, int argc, char *argv[])
{
notmuch_config_t *config;
notmuch_database_t *notmuch;
- add_files_state_t add_files_state;
+ struct priv_ctx state2;
+ notmuch_indexing_context_t add_files_state; /* FIXME: Rename */
double elapsed;
struct timeval tv_now;
int ret = 0;
@@ -715,13 +199,14 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
const char *db_path;
char *dot_notmuch_path;
struct sigaction action;
- _filename_node_t *f;
- int renamed_files, removed_files;
- notmuch_status_t status;
int i;
+ notmuch_mailstore_t *mailstore;
+
+ indexing_ctx = &add_files_state;
add_files_state.verbose = 0;
- add_files_state.output_is_a_tty = isatty (fileno (stdout));
+ add_files_state.priv = &state2;
+ state2.output_is_a_tty = isatty (fileno (stdout));
for (i = 0; i < argc && argv[i][0] == '-'; i++) {
if (STRNCMP_LITERAL (argv[i], "--verbose") == 0) {
@@ -737,6 +222,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
return 1;
db_path = notmuch_config_get_database_path (config);
+ mailstore = notmuch_config_get_mailstore (config);
dot_notmuch_path = talloc_asprintf (ctx, "%s/%s", db_path, ".notmuch");
@@ -744,24 +230,22 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
int count;
count = 0;
- count_files (db_path, &count);
+ notmuch_mailstore_count_files (mailstore, db_path, &count, &interrupted);
if (interrupted)
return 1;
printf ("Found %d total files (that's not much mail).\n", count);
- notmuch = notmuch_database_create (db_path,
- notmuch_config_get_mailstore (config));
+ notmuch = notmuch_database_create (db_path, mailstore);
add_files_state.total_files = count;
} else {
- notmuch = notmuch_database_open (db_path,
- NOTMUCH_DATABASE_MODE_READ_WRITE,
- notmuch_config_get_mailstore (config));
+ notmuch = notmuch_database_open (db_path, NOTMUCH_DATABASE_MODE_READ_WRITE,
+ mailstore);
if (notmuch == NULL)
return 1;
if (notmuch_database_needs_upgrade (notmuch)) {
printf ("Welcome to a new version of notmuch! Your database will now be upgraded.\n");
- gettimeofday (&add_files_state.tv_start, NULL);
+ gettimeofday (&state2.tv_start, NULL);
notmuch_database_upgrade (notmuch, upgrade_print_progress,
&add_files_state);
printf ("Your notmuch database has now been upgraded to database format version %u.\n",
@@ -777,6 +261,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
/* Setup our handler for SIGINT. We do this after having
* potentially done a database upgrade we this interrupt handler
* won't support. */
+ add_files_state.interrupted = 0;
memset (&action, 0, sizeof (struct sigaction));
action.sa_handler = handle_sigint;
sigemptyset (&action.sa_mask);
@@ -788,33 +273,12 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
add_files_state.processed_files = 0;
add_files_state.added_messages = 0;
- gettimeofday (&add_files_state.tv_start, NULL);
-
- add_files_state.removed_files = _filename_list_create (ctx);
- add_files_state.removed_directories = _filename_list_create (ctx);
-
- ret = add_files (notmuch, db_path, &add_files_state);
-
- removed_files = 0;
- renamed_files = 0;
- for (f = add_files_state.removed_files->head; f; f = f->next) {
- status = notmuch_database_remove_message (notmuch, f->filename);
- if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID)
- renamed_files++;
- else
- removed_files++;
- }
-
- for (f = add_files_state.removed_directories->head; f; f = f->next) {
- _remove_directory (ctx, notmuch, f->filename,
- &renamed_files, &removed_files);
- }
+ gettimeofday (&state2.tv_start, NULL);
- talloc_free (add_files_state.removed_files);
- talloc_free (add_files_state.removed_directories);
+ ret = add_files (mailstore, db_path, &add_files_state);
gettimeofday (&tv_now, NULL);
- elapsed = notmuch_time_elapsed (add_files_state.tv_start,
+ elapsed = notmuch_time_elapsed (state2.tv_start,
tv_now);
if (add_files_state.processed_files) {
@@ -839,16 +303,16 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
printf ("No new mail.");
}
- if (removed_files) {
+ if (add_files_state.removed_files) {
printf (" Removed %d %s.",
- removed_files,
- removed_files == 1 ? "message" : "messages");
+ add_files_state.removed_files,
+ add_files_state.removed_files == 1 ? "message" : "messages");
}
- if (renamed_files) {
+ if (add_files_state.renamed_files) {
printf (" Detected %d file %s.",
- renamed_files,
- renamed_files == 1 ? "rename" : "renames");
+ add_files_state.renamed_files,
+ add_files_state.renamed_files == 1 ? "rename" : "renames");
}
printf ("\n");
--
1.7.0.2
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [PATCH v3 3/6] Add maildir-based mailstore
2010-03-27 20:42 ` Michal Sojka
2010-03-27 20:44 ` [PATCH v3 1/6] Mailstore abstraction interface Michal Sojka
2010-03-27 20:44 ` [PATCH v3 2/6] Conversion to mailstore abstraction Michal Sojka
@ 2010-03-27 20:44 ` Michal Sojka
2010-03-27 20:44 ` [PATCH v3 4/6] Tests for " Michal Sojka
` (3 subsequent siblings)
6 siblings, 0 replies; 25+ messages in thread
From: Michal Sojka @ 2010-03-27 20:44 UTC (permalink / raw)
To: notmuch
This mailstore allows bi-directional synchronization between maildir
flags and certain tags. The flag-to-tag mapping is defined by flag2tag
array.
The synchronization works this way:
1) Whenever notmuch new is executed, the following happens:
o New messages are tagged with inbox.
o New messages without maildir info in the file name (typically
files in new/) are tagged with unread.
o For new or renamed messages with maildir info present in the
file name, the tags defined in flag2tag are either added or
removed depending on the flags from the file name.
2) Whenever notmuch tag (or notmuch restore) is executed, a new set of
flags based on the tags is constructed for every message and a new
file name is prepared based on the old file name but with the new
flags. If the old message file was in 'new' directory then this is
replaced with 'cur' in the new file name. If the new and old file
names differ, the file is renamed and notmuch database is updated
accordingly.
The rename happens before the database is updated. In case of crash
between rename and database update, the next run of notmuch new
brings the database in sync with the mail store again.
This mailstore is enabled by putting the following to your
.notmuch-config:
[mailstore]
type=maildir
Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz>
---
lib/database.cc | 7 ++
lib/mailstore-files.c | 203 ++++++++++++++++++++++++++++++++++++++++++++++++-
lib/mailstore.c | 1 +
lib/message.cc | 41 ++++++++++-
lib/notmuch-private.h | 4 +
lib/notmuch.h | 1 +
6 files changed, 253 insertions(+), 4 deletions(-)
diff --git a/lib/database.cc b/lib/database.cc
index 93c8d0f..33ef889 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1471,6 +1471,13 @@ notmuch_database_add_message (notmuch_database_t *notmuch,
_notmuch_message_add_filename (message, filename);
+ /* This is a new message or it has a new filename and as such,
+ * its tags in database either do not exists or might be out
+ * of date. Mailstore assigns the tags later in index_new(),
+ * but until then we should not synchronize the tags back to
+ * the mailstore. */
+ notmuch_message_set_flag(message, NOTMUCH_MESSAGE_FLAG_TAGS_INVALID, TRUE);
+
/* Is this a newly created message object? */
if (private_status == NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND) {
_notmuch_message_add_term (message, "type", "mail");
diff --git a/lib/mailstore-files.c b/lib/mailstore-files.c
index e4086ba..b4e6e42 100644
--- a/lib/mailstore-files.c
+++ b/lib/mailstore-files.c
@@ -24,6 +24,30 @@
#include "notmuch.h"
#include "mailstore-private.h"
#include <dirent.h>
+#include <stdbool.h>
+
+#define ARRAY_SIZE(arr) (sizeof (arr) / sizeof (arr[0]))
+
+struct mailstore_priv {
+ void (*tag_new)(notmuch_message_t *message, const char *filename);
+ int (*tag_renamed)(notmuch_message_t *message, const char *filename);
+};
+
+struct maildir_flag_tag {
+ char flag;
+ const char *tag;
+ bool inverse;
+};
+
+/* ASCII ordered table of Maildir flags and assiciated tags */
+struct maildir_flag_tag flag2tag[] = {
+ { 'D', "draft", false},
+ { 'F', "flagged", false},
+ { 'P', "passed", false},
+ { 'R', "replied", false},
+ { 'S', "unread", true },
+ { 'T', "delete", false},
+};
typedef struct _filename_node {
char *filename;
@@ -69,8 +93,9 @@ _filename_list_add (_filename_list_t *list,
}
static void
-tag_inbox_and_unread (notmuch_message_t *message)
+tag_inbox_and_unread (notmuch_message_t *message, const char *filename)
{
+ (void)filename;
notmuch_message_add_tag (message, "inbox");
notmuch_message_add_tag (message, "unread");
}
@@ -116,6 +141,156 @@ _entries_resemble_maildir (struct dirent **entries, int count)
return 0;
}
+static int
+tag_from_maildir_flags (notmuch_message_t *message, const char *filename)
+{
+ const char *flags, *p;
+ char f;
+ bool valid, unread;
+ unsigned i;
+
+ flags = strstr(filename, ":2,");
+ if (!flags)
+ return -1;
+ flags += 3;
+
+ /* Check that the letters are valid Maildir flags */
+ f = 0;
+ valid = true;
+ for (p=flags; valid && *p; p++) {
+ switch (*p) {
+ case 'P':
+ case 'R':
+ case 'S':
+ case 'T':
+ case 'D':
+ case 'F':
+ if (*p > f) f=*p;
+ else valid = false;
+ break;
+ default:
+ valid = false;
+ }
+ }
+ if (!valid) {
+ fprintf (stderr, "Warning: Invalid maildir flags in filename %s\n", filename);
+ return -1;
+ }
+
+ notmuch_message_freeze(message);
+ unread = true;
+ for (i = 0; i < ARRAY_SIZE(flag2tag); i++) {
+ if ((strchr(flags, flag2tag[i].flag) != NULL) ^ flag2tag[i].inverse) {
+ notmuch_message_add_tag (message, flag2tag[i].tag);
+ } else {
+ notmuch_message_remove_tag (message, flag2tag[i].tag);
+ }
+ }
+ notmuch_message_thaw(message);
+
+ /* From now on, we can synchronize the tags from the database to
+ * the mailstore. */
+ notmuch_message_set_flag(message, NOTMUCH_MESSAGE_FLAG_TAGS_INVALID, FALSE);
+ return 0;
+}
+
+static void
+get_new_flags(notmuch_message_t *message, char *flags)
+{
+ notmuch_tags_t *tags;
+ const char *tag;
+ unsigned i;
+ char *p;
+
+ for (i = 0; i < ARRAY_SIZE(flag2tag); i++)
+ flags[i] = flag2tag[i].inverse ? flag2tag[i].flag : '\0';
+
+ for (tags = notmuch_message_get_tags (message);
+ notmuch_tags_valid (tags);
+ notmuch_tags_move_to_next (tags))
+ {
+ tag = notmuch_tags_get (tags);
+ for (i = 0; i < ARRAY_SIZE(flag2tag); i++) {
+ if (strcmp(tag, flag2tag[i].tag) == 0)
+ flags[i] = flag2tag[i].inverse ? '\0' : flag2tag[i].flag;
+ }
+ }
+
+ p = flags;
+ for (i = 0; i < ARRAY_SIZE(flag2tag); i++) {
+ if (flags[i])
+ *p++ = flags[i];
+ }
+ *p = '\0';
+}
+
+static char *
+get_subdir (char *filename)
+{
+ char *p, *subdir = NULL;
+
+ p = filename + strlen (filename) - 1;
+ while (p > filename + 3 && *p != '/')
+ p--;
+ if (*p == '/') {
+ subdir = p - 3;
+ if (subdir > filename && *(subdir - 1) != '/')
+ subdir = NULL;
+ }
+ return subdir;
+}
+
+/* Store maildir-related tags as maildir flags */
+static notmuch_private_status_t
+maildir_sync_tags (notmuch_mailstore_t *mailstore,
+ notmuch_message_t *message)
+{
+ char flags[ARRAY_SIZE(flag2tag)+1];
+ const char *filename, *p;
+ char *filename_new, *subdir = NULL;
+ int ret;
+
+ (void)mailstore;
+ get_new_flags (message, flags);
+
+ filename = notmuch_message_get_filename (message);
+ p = strstr(filename, ":2,");
+ if (!p)
+ p = filename + strlen(filename);
+
+ filename_new = talloc_size(message, (p-filename) + 3 + sizeof(flags));
+ if (unlikely (filename_new == NULL))
+ return NOTMUCH_STATUS_OUT_OF_MEMORY;
+ memcpy(filename_new, filename, p-filename);
+ filename_new[p-filename] = '\0';
+
+ /* If message is in new/ move it under cur/. */
+ subdir = get_subdir (filename_new);
+ if (subdir && memcmp (subdir, "new/", 4) == 0)
+ memcpy (subdir, "cur/", 4);
+
+ strcpy (filename_new+(p-filename), ":2,");
+ strcpy (filename_new+(p-filename)+3, flags);
+
+ if (strcmp (filename, filename_new) != 0) {
+ ret = rename (filename, filename_new);
+ if (ret == -1) {
+ perror (talloc_asprintf (message, "rename of %s to %s failed", filename, filename_new));
+ exit (1);
+ }
+ return _notmuch_message_rename (message, filename_new);
+ /* _notmuch_message_sync is our caller. Do not call it here. */
+ }
+ return NOTMUCH_STATUS_SUCCESS;
+}
+
+static void
+tag_inbox_and_maildir_flags (notmuch_message_t *message, const char *filename)
+{
+ notmuch_message_add_tag (message, "inbox");
+ if (tag_from_maildir_flags(message, filename) != 0)
+ notmuch_message_add_tag (message, "unread");
+}
/* Examine 'path' recursively as follows:
*
@@ -171,6 +346,7 @@ add_files_recursive (notmuch_mailstore_t *mailstore,
struct stat st;
notmuch_bool_t is_maildir, new_directory;
_indexing_context_priv_t *priv = state->priv;
+ struct mailstore_priv *mailstore_priv = mailstore->priv;
notmuch_database_t *notmuch = mailstore->notmuch;
if (stat (path, &st)) {
@@ -352,11 +528,12 @@ add_files_recursive (notmuch_mailstore_t *mailstore,
/* success */
case NOTMUCH_STATUS_SUCCESS:
state->added_messages++;
- tag_inbox_and_unread (message);
+ mailstore_priv->tag_new (message, next);
break;
/* Non-fatal issues (go on to next file) */
case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
- /* Stay silent on this one. */
+ if (mailstore_priv->tag_renamed)
+ mailstore_priv->tag_renamed (message, next);
break;
case NOTMUCH_STATUS_FILE_NOT_EMAIL:
fprintf (stderr, "Note: Ignoring non-mail file: %s\n",
@@ -614,6 +791,10 @@ open_file(notmuch_mailstore_t *mailstore, const char *filename)
return fopen (abs_filename, "r");
}
+struct mailstore_priv files_priv = {
+ .tag_new = tag_inbox_and_unread,
+};
+
/* Original notmuch mail store */
struct _notmuch_mailstore mailstore_files = {
.type = "files",
@@ -621,4 +802,20 @@ struct _notmuch_mailstore mailstore_files = {
.index_new = index_new,
.sync_tags = NULL,
.open_file = open_file,
+ .priv = &files_priv,
+};
+
+struct mailstore_priv maildir_priv = {
+ .tag_new = tag_inbox_and_maildir_flags,
+ .tag_renamed = tag_from_maildir_flags,
+};
+
+/* Similar to mailstore_files, but does bi-directional synchronization between certain tags and maildir flags */
+struct _notmuch_mailstore mailstore_maildir = {
+ .type = "maildir",
+ .count_files = count_files,
+ .index_new = index_new,
+ .sync_tags = maildir_sync_tags,
+ .open_file = open_file,
+ .priv = &maildir_priv,
};
diff --git a/lib/mailstore.c b/lib/mailstore.c
index cdac9a6..7dc55ad 100644
--- a/lib/mailstore.c
+++ b/lib/mailstore.c
@@ -25,6 +25,7 @@
static notmuch_mailstore_t *available[] = {
&mailstore_files,
+ &mailstore_maildir,
};
notmuch_mailstore_t *
diff --git a/lib/message.cc b/lib/message.cc
index c32ee7d..51208ba 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -561,7 +561,8 @@ _notmuch_message_sync (notmuch_message_t *message)
if (message->notmuch->mode == NOTMUCH_DATABASE_MODE_READ_ONLY)
return;
- if (message->notmuch->mailstore->sync_tags) {
+ if (message->notmuch->mailstore->sync_tags &&
+ !notmuch_message_get_flag(message, NOTMUCH_MESSAGE_FLAG_TAGS_INVALID)) {
status = message->notmuch->mailstore->sync_tags (message->notmuch->mailstore,
message);
if (status != NOTMUCH_PRIVATE_STATUS_SUCCESS) {
@@ -683,6 +684,44 @@ _notmuch_message_remove_term (notmuch_message_t *message,
return NOTMUCH_PRIVATE_STATUS_SUCCESS;
}
+/* Change the message filename stored in the database.
+ *
+ * This change will not be reflected in the database until the next
+ * call to _notmuch_message_sync.
+ */
+notmuch_private_status_t
+_notmuch_message_rename (notmuch_message_t *message,
+ const char *new_filename)
+{
+ void *local = talloc_new (message);
+ char *direntry;
+ Xapian::PostingIterator i, end;
+ Xapian::Document document;
+ notmuch_private_status_t pstatus;
+ notmuch_status_t status;
+ const char *old_filename;
+
+ old_filename = notmuch_message_get_filename(message);
+ old_filename = talloc_reference(local, old_filename);
+ if (unlikely(!old_filename))
+ return NOTMUCH_PRIVATE_STATUS_OUT_OF_MEMORY;
+
+ status = _notmuch_message_add_filename (message, new_filename);
+ if (status)
+ return (notmuch_private_status_t)status;
+
+ status = _notmuch_database_filename_to_direntry (local, message->notmuch,
+ old_filename, &direntry);
+ if (status)
+ return (notmuch_private_status_t)status;
+
+ pstatus = _notmuch_message_remove_term (message, "file-direntry", direntry);
+
+ talloc_free (local);
+
+ return pstatus;
+}
+
notmuch_status_t
notmuch_message_add_tag (notmuch_message_t *message, const char *tag)
{
diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index d52d84d..63e75ec 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -245,6 +245,10 @@ notmuch_status_t
_notmuch_message_add_filename (notmuch_message_t *message,
const char *filename);
+notmuch_private_status_t
+_notmuch_message_rename (notmuch_message_t *message,
+ const char *new_filename);
+
void
_notmuch_message_ensure_thread_id (notmuch_message_t *message);
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 8fa7b31..c554f6d 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -760,6 +760,7 @@ notmuch_message_get_filename (notmuch_message_t *message);
/* Message flags */
typedef enum _notmuch_message_flag {
NOTMUCH_MESSAGE_FLAG_MATCH,
+ NOTMUCH_MESSAGE_FLAG_TAGS_INVALID,
} notmuch_message_flag_t;
/* Get a value of a flag for the email corresponding to 'message'. */
--
1.7.0.2
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [PATCH v3 4/6] Tests for maildir-based mailstore
2010-03-27 20:42 ` Michal Sojka
` (2 preceding siblings ...)
2010-03-27 20:44 ` [PATCH v3 3/6] Add maildir-based mailstore Michal Sojka
@ 2010-03-27 20:44 ` Michal Sojka
2010-03-27 20:44 ` [PATCH v3 5/6] Access messages through mail store interface Michal Sojka
` (2 subsequent siblings)
6 siblings, 0 replies; 25+ messages in thread
From: Michal Sojka @ 2010-03-27 20:44 UTC (permalink / raw)
To: notmuch
Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz>
---
test/t0006-maildir.sh | 208 +++++++++++++++++++++++++++++++++++++++++++++++++
test/test-lib.sh | 7 +-
2 files changed, 213 insertions(+), 2 deletions(-)
create mode 100755 test/t0006-maildir.sh
diff --git a/test/t0006-maildir.sh b/test/t0006-maildir.sh
new file mode 100755
index 0000000..03a2a35
--- /dev/null
+++ b/test/t0006-maildir.sh
@@ -0,0 +1,208 @@
+#!/bin/bash
+
+test_description="Test maildir mailstore"
+
+. ./test-lib.sh
+
+filter_output() {
+ grep -v -E -e "$NOTMUCH_IGNORED_OUTPUT_REGEXP" | sed -e "$NOTMUCH_THREAD_ID_SQUELCH"
+}
+
+filter_show() {
+ sed -e 's/, /,\n/g'|sed -e "s|${MAIL_DIR}/||" -e '/^"tags"/d'
+ echo
+}
+
+cat >> "$NOTMUCH_CONFIG" <<EOF
+[mailstore]
+type=maildir
+EOF
+
+test_expect_success "No new mail" '
+execute_expecting new "No new mail."
+'
+cat > expected <<EOF
+Added 1 new message to the database.
+EOF
+test_expect_success "Add a message, no flags" '
+generate_message [subject]="\"test message\"" [date]="\"Sat, 01 Jan 2000 12:00:00 -0000\"" [filename]="\"msg-001:2,\"" &&
+notmuch new | filter_output > actual &&
+test_cmp expected actual
+#emacs --eval "(gdb \"gdb --annotate=3 --args $(which notmuch) new\")"
+'
+cat > expected <<EOF
+thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test message (inbox unread)
+EOF
+test_expect_success 'Search for the message' '
+notmuch search tag:inbox and tag:unread | filter_output > actual &&
+test_cmp expected actual
+'
+cat > expected <<EOF
+No new mail. Detected 1 file rename.
+EOF
+test_expect_success 'Add seen flag' '
+mv "${gen_msg_filename}" "${gen_msg_filename}S" &&
+increment_mtime "$(dirname "${gen_msg_filename}")" &&
+notmuch new | filter_output > actual &&
+test_cmp expected actual
+'
+cat > expected <<EOF
+thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test message (inbox)
+EOF
+test_expect_success 'Check that tags were updated' '
+notmuch search tag:inbox and not tag:unread | filter_output > actual &&
+test_cmp expected actual
+'
+cat > expected <<EOF
+Added 1 new message to the database.
+EOF
+test_expect_success "Add a seen message" '
+generate_message [subject]="\"test message 2\"" [date]="\"Sat, 01 Jan 2000 12:00:00 -0000\"" [filename]="\"msg-002:2,S\"" &&
+notmuch new | filter_output > actual &&
+test_cmp expected actual
+'
+cat > expected <<EOF
+thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test message (inbox)
+thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test message 2 (inbox)
+EOF
+test_expect_success 'Check that the seen message is not tagged unread' '
+notmuch search tag:inbox and not tag:unread | filter_output > actual &&
+test_cmp expected actual
+'
+test_expect_success 'Tag the seen messages as replied' '
+notmuch tag +replied -inbox tag:inbox and not tag:unread
+'
+
+cat > expected <<EOF
+msg-001:2,RS
+msg-002:2,RS
+EOF
+test_expect_success 'Check that R flag was added' '
+ls -1 "${MAIL_DIR}" > actual &&
+test_cmp expected actual
+'
+cat <<EOF > show-expected
+[[[{"id": "msg-001@notmuch-test-suite",
+"match": true,
+"filename": "msg-001:2,RS",
+"headers": {"Subject": "test message",
+"From": "Notmuch Test Suite <test_suite@notmuchmail.org>",
+"To": "Notmuch Test Suite <test_suite@notmuchmail.org>",
+"Cc": "",
+"Bcc": "",
+"Date": "Sat,
+01 Jan 2000 12:00:00 -0000"},
+"body": [{"id": 1,
+"content-type": "text/plain",
+"content": "This is just a test message at msg-001:2,\n"}]},
+[]]]]
+EOF
+
+test_expect_success 'Renamed message can be shown without running notmuch new' '
+notmuch show --format=json id:msg-001@notmuch-test-suite | filter_show > show-actual &&
+test_cmp show-expected show-actual
+'
+
+test_expect_success 'Test that we can reply to the renamed message' '
+notmuch reply id:msg-001@notmuch-test-suite
+'
+
+echo "No new mail." > expected
+test_expect_success 'No rename should be detected by notmuch new' '
+increment_mtime "$(dirname "${gen_msg_filename}")" &&
+notmuch new > actual &&
+test_cmp expected actual
+'
+test_expect_success "Add a message to new/ without info" '
+generate_message [subject]="\"test message 3\"" [date]="\"Sat, 01 Jan 2000 12:00:00 -0000\"" [dir]=new &&
+notmuch new | filter_output > actual &&
+test_cmp - actual <<EOF
+Added 1 new message to the database.
+EOF
+'
+test_expect_success "Check that the message has inbox and unread tags" '
+notmuch search tag:inbox and tag:unread | filter_output > actual &&
+test_cmp - actual <<EOF
+thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test message 3 (inbox unread)
+EOF
+'
+test_expect_success "Check that the message was not renamed" '
+ls "${MAIL_DIR}/new" > actual &&
+test_cmp - actual <<EOF
+msg-003
+EOF
+'
+test_expect_success 'Removing of unread tag should fail without cur/' '
+test_must_fail notmuch tag -unread tag:inbox and tag:unread
+'
+test_expect_success "Check that the tags were not changed" '
+notmuch search tag:inbox and tag:unread | filter_output > actual &&
+test_cmp - actual <<EOF
+thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test message 3 (inbox unread)
+EOF
+'
+
+# notmuch new is not necessary here, but we run it in order to check
+# for 'no rename' later (*).
+test_expect_success 'Create cur/ and let notmuch know about it' '
+mkdir "$MAIL_DIR/cur" &&
+notmuch new
+'
+test_expect_success 'Removing of unread tag should pass with cur/' '
+notmuch tag -unread tag:inbox and tag:unread
+'
+test_expect_success 'Check that the message was moved to cur/' '\
+ls "$MAIL_DIR/cur" > actual &&
+test_cmp - actual <<EOF
+msg-003:2,S
+EOF
+'
+test_expect_success 'No rename should be detected by notmuch new' '
+increment_mtime "$MAIL_DIR/cur" &&
+notmuch new > actual &&
+test_cmp - actual <<EOF
+No new mail.
+EOF
+'
+# (*) If notmuch new was not run we've got "Processed 1 file in almost
+# no time" here. The reason is that removing unread tag in a previous
+# test created directory document in the database but this document
+# was not linked as subdirectory of $MAIL_DIR. Therefore notmuch new
+# could not reach the cur/ directory and its files in it during
+# recurive traversal.
+test_expect_success 'Remove info from file name' '
+mv "$MAIL_DIR/cur/msg-003:2,S" "$MAIL_DIR/cur/msg-003" &&
+increment_mtime "$MAIL_DIR/cur" &&
+notmuch new | filter_output > actual
+test_cmp - actual <<EOF
+No new mail. Detected 1 file rename.
+EOF
+'
+test_expect_success "Check that removing info did not change tags" '
+notmuch search tag:inbox | filter_output > actual &&
+test_cmp - actual <<EOF
+thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test message 3 (inbox)
+EOF
+'
+test_expect_success "Add a message to fakenew/ without info" '
+generate_message [subject]="\"test message 4\"" [date]="\"Sat, 01 Jan 2000 12:00:00 -0000\"" [dir]=fakenew &&
+notmuch new | filter_output > actual &&
+test_cmp - actual <<EOF
+Added 1 new message to the database.
+EOF
+'
+test_expect_success "Check that the message has inbox and unread tags" '
+notmuch search tag:inbox and tag:unread | filter_output > actual &&
+test_cmp - actual <<EOF
+thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; test message 4 (inbox unread)
+EOF
+'
+test_expect_success 'Removing of unread tag should leave the message in fakenew/' '
+notmuch tag -unread tag:inbox and tag:unread &&
+ls "$MAIL_DIR/fakenew" > actual &&
+test_cmp - actual <<EOF
+msg-004:2,S
+EOF
+'
+
+test_done
diff --git a/test/test-lib.sh b/test/test-lib.sh
index 5417fe7..917631b 100755
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -257,8 +257,11 @@ generate_message ()
local additional_headers
gen_msg_cnt=$((gen_msg_cnt + 1))
- gen_msg_name=msg-$(printf "%03d" $gen_msg_cnt)
- gen_msg_id="${gen_msg_name}@notmuch-test-suite"
+ if [ -z "${template[filename]}" ]; then
+ template[filename]="msg-$(printf "%03d" $gen_msg_cnt)"
+ fi
+ gen_msg_name=${template[filename]}
+ gen_msg_id="${gen_msg_name%:2,*}@notmuch-test-suite"
if [ -z "${template[dir]}" ]; then
gen_msg_filename="${MAIL_DIR}/$gen_msg_name"
--
1.7.0.2
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [PATCH v3 5/6] Access messages through mail store interface
2010-03-27 20:42 ` Michal Sojka
` (3 preceding siblings ...)
2010-03-27 20:44 ` [PATCH v3 4/6] Tests for " Michal Sojka
@ 2010-03-27 20:44 ` Michal Sojka
2010-03-27 20:44 ` [PATCH v3 6/6] Add 'cat' subcommand Michal Sojka
2010-03-27 20:49 ` Mailstore abstraction & maildir synchronization Michal Sojka
6 siblings, 0 replies; 25+ messages in thread
From: Michal Sojka @ 2010-03-27 20:44 UTC (permalink / raw)
To: notmuch
This patch modifies notmuch binary to access the messages through mail
store interface, so that non-file based mail stores can also be
implemented.
The API of notmuch library was changed. Now,
notmuch_message_get_filename() returns relative file name with respect
to the database path. As a result, notmuch show also outputs relative
paths so that MUAs need to be changed.
Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz>
---
lib/database.cc | 14 +++++++++++---
lib/index.cc | 8 ++++++--
lib/mailstore-files.c | 24 +++++++++++++++++-------
lib/message-file.c | 8 ++++----
lib/message.cc | 33 +++++++++++++++++++++++++--------
lib/notmuch-private.h | 6 +++---
lib/notmuch.h | 16 ++++++++++++++--
lib/sha1.c | 6 +-----
notmuch-client.h | 2 +-
notmuch-reply.c | 10 +++++++++-
notmuch-show.c | 14 ++++++++++++--
show-message.c | 14 +-------------
12 files changed, 104 insertions(+), 51 deletions(-)
diff --git a/lib/database.cc b/lib/database.cc
index 33ef889..4f3ce88 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1374,6 +1374,7 @@ notmuch_database_add_message (notmuch_database_t *notmuch,
notmuch_message_t *message = NULL;
notmuch_status_t ret = NOTMUCH_STATUS_SUCCESS;
notmuch_private_status_t private_status;
+ const char *relative;
const char *date, *header;
const char *from, *to, *subject;
@@ -1386,7 +1387,8 @@ notmuch_database_add_message (notmuch_database_t *notmuch,
if (ret)
return ret;
- message_file = notmuch_message_file_open (filename);
+ relative = _notmuch_database_relative_path (notmuch, filename);
+ message_file = notmuch_message_file_open (notmuch->mailstore, relative);
if (message_file == NULL)
return NOTMUCH_STATUS_FILE_ERROR;
@@ -1438,9 +1440,15 @@ notmuch_database_add_message (notmuch_database_t *notmuch,
}
if (message_id == NULL ) {
+ FILE *file;
+ char *sha1 = NULL;
/* No message-id at all, let's generate one by taking a
* hash over the file's contents. */
- char *sha1 = notmuch_sha1_of_file (filename);
+ file = notmuch_mailstore_open_file (notmuch->mailstore, relative);
+ if (file) {
+ sha1 = notmuch_sha1_of_file (file);
+ fclose (file);
+ }
/* If that failed too, something is really wrong. Give up. */
if (sha1 == NULL) {
@@ -1490,7 +1498,7 @@ notmuch_database_add_message (notmuch_database_t *notmuch,
date = notmuch_message_file_get_header (message_file, "date");
_notmuch_message_set_date (message, date);
- _notmuch_message_index_file (message, filename);
+ _notmuch_message_index_file (message, relative);
} else {
ret = NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID;
}
diff --git a/lib/index.cc b/lib/index.cc
index cf93025..4d8c4dd 100644
--- a/lib/index.cc
+++ b/lib/index.cc
@@ -425,15 +425,19 @@ _notmuch_message_index_file (notmuch_message_t *message,
const char *from, *subject;
notmuch_status_t ret = NOTMUCH_STATUS_SUCCESS;
static int initialized = 0;
+ notmuch_mailstore_t *mailstore;
if (! initialized) {
g_mime_init (0);
initialized = 1;
}
- file = fopen (filename, "r");
+ mailstore = notmuch_message_get_mailstore(message);
+ file = notmuch_mailstore_open_file (mailstore, filename);
if (! file) {
- fprintf (stderr, "Error opening %s: %s\n", filename, strerror (errno));
+ fprintf (stderr, "Error opening %s: %s\n",
+ notmuch_message_get_filename (message),
+ strerror (errno));
ret = NOTMUCH_STATUS_FILE_ERROR;
goto DONE;
}
diff --git a/lib/mailstore-files.c b/lib/mailstore-files.c
index b4e6e42..0250dba 100644
--- a/lib/mailstore-files.c
+++ b/lib/mailstore-files.c
@@ -246,9 +246,10 @@ maildir_sync_tags (notmuch_mailstore_t *mailstore,
notmuch_message_t *message)
{
char flags[ARRAY_SIZE(flag2tag)+1];
- const char *filename, *p;
+ const char *filename, *p, *db_path;
char *filename_new, *subdir = NULL;
int ret;
+ char *abs1, *abs2;
(void)mailstore;
get_new_flags (message, flags);
@@ -273,11 +274,16 @@ maildir_sync_tags (notmuch_mailstore_t *mailstore,
strcpy (filename_new+(p-filename)+3, flags);
if (strcmp (filename, filename_new) != 0) {
- ret = rename (filename, filename_new);
+ db_path = notmuch_database_get_path (mailstore->notmuch);
+ asprintf(&abs1, "%s/%s", db_path, filename);
+ asprintf(&abs2, "%s/%s", db_path, filename_new);
+ ret = rename (abs1, abs2);
if (ret == -1) {
- perror (talloc_asprintf (message, "rename of %s to %s failed", filename, filename_new));
+ perror (talloc_asprintf (message, "rename of %s to %s failed", abs1, abs2));
exit (1);
}
+ free(abs1);
+ free(abs2);
return _notmuch_message_rename (message, filename_new);
/* _notmuch_message_sync is our caller. Do not call it here. */
}
@@ -782,13 +788,17 @@ index_new(notmuch_mailstore_t *mailstore, const char* path,
static FILE *
open_file(notmuch_mailstore_t *mailstore, const char *filename)
{
- const char *db_path, *abs_filename;
-
+ const char *db_path;
+ char *abs_filename;
+ FILE *file;
+
db_path = notmuch_database_get_path(mailstore->notmuch);
- abs_filename = talloc_asprintf(mailstore, "%s/%s", db_path, filename);
+ abs_filename = talloc_asprintf(NULL, "%s/%s", db_path, filename);
if (unlikely(abs_filename == NULL))
return NULL;
- return fopen (abs_filename, "r");
+ file = fopen (abs_filename, "r");
+ talloc_free(abs_filename);
+ return file;
}
struct mailstore_priv files_priv = {
diff --git a/lib/message-file.c b/lib/message-file.c
index 3a1a681..5752e69 100644
--- a/lib/message-file.c
+++ b/lib/message-file.c
@@ -94,7 +94,7 @@ _notmuch_message_file_destructor (notmuch_message_file_t *message)
/* Create a new notmuch_message_file_t for 'filename' with 'ctx' as
* the talloc owner. */
notmuch_message_file_t *
-_notmuch_message_file_open_ctx (void *ctx, const char *filename)
+_notmuch_message_file_open_ctx (void *ctx, notmuch_mailstore_t *mailstore, const char *filename)
{
notmuch_message_file_t *message;
@@ -104,7 +104,7 @@ _notmuch_message_file_open_ctx (void *ctx, const char *filename)
talloc_set_destructor (message, _notmuch_message_file_destructor);
- message->file = fopen (filename, "r");
+ message->file = notmuch_mailstore_open_file(mailstore, filename);
if (message->file == NULL)
goto FAIL;
@@ -126,9 +126,9 @@ _notmuch_message_file_open_ctx (void *ctx, const char *filename)
}
notmuch_message_file_t *
-notmuch_message_file_open (const char *filename)
+notmuch_message_file_open (notmuch_mailstore_t *mailstore, const char *filename)
{
- return _notmuch_message_file_open_ctx (NULL, filename);
+ return _notmuch_message_file_open_ctx (NULL, mailstore, filename);
}
void
diff --git a/lib/message.cc b/lib/message.cc
index 51208ba..292e95e 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -247,6 +247,7 @@ static void
_notmuch_message_ensure_message_file (notmuch_message_t *message)
{
const char *filename;
+ notmuch_mailstore_t *mailstore;
if (message->message_file)
return;
@@ -255,7 +256,9 @@ _notmuch_message_ensure_message_file (notmuch_message_t *message)
if (unlikely (filename == NULL))
return;
- message->message_file = _notmuch_message_file_open_ctx (message, filename);
+ mailstore = notmuch_message_get_mailstore (message);
+
+ message->message_file = _notmuch_message_file_open_ctx (message, mailstore, filename);
}
const char *
@@ -429,7 +432,7 @@ notmuch_message_get_filename (notmuch_message_t *message)
int prefix_len = strlen (prefix);
Xapian::TermIterator i;
char *colon, *direntry = NULL;
- const char *db_path, *directory, *basename;
+ const char *directory, *basename;
unsigned int directory_id;
void *local = talloc_new (message);
@@ -474,18 +477,16 @@ notmuch_message_get_filename (notmuch_message_t *message)
*colon = '\0';
- db_path = notmuch_database_get_path (message->notmuch);
-
directory = _notmuch_database_get_directory_path (local,
message->notmuch,
directory_id);
if (strlen (directory))
- message->filename = talloc_asprintf (message, "%s/%s/%s",
- db_path, directory, basename);
- else
message->filename = talloc_asprintf (message, "%s/%s",
- db_path, basename);
+ directory, basename);
+ else
+ message->filename = talloc_asprintf (message, "%s",
+ basename);
talloc_free ((void *) directory);
talloc_free (local);
@@ -493,6 +494,22 @@ notmuch_message_get_filename (notmuch_message_t *message)
return message->filename;
}
+FILE *
+notmuch_message_fopen (notmuch_message_t *message)
+{
+ const char *filename;
+ filename = notmuch_message_get_filename (message);
+ return notmuch_mailstore_open_file (message->notmuch->mailstore,
+ filename);
+}
+
+notmuch_mailstore_t *
+notmuch_message_get_mailstore (notmuch_message_t *message)
+{
+ return message->notmuch->mailstore;
+}
+
+
notmuch_bool_t
notmuch_message_get_flag (notmuch_message_t *message,
notmuch_message_flag_t flag)
diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index 63e75ec..c257365 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -299,11 +299,11 @@ typedef struct _notmuch_message_file notmuch_message_file_t;
* Returns NULL if any error occurs.
*/
notmuch_message_file_t *
-notmuch_message_file_open (const char *filename);
+notmuch_message_file_open (notmuch_mailstore_t *mailstore, const char *filename);
/* Like notmuch_message_file_open but with 'ctx' as the talloc owner. */
notmuch_message_file_t *
-_notmuch_message_file_open_ctx (void *ctx, const char *filename);
+_notmuch_message_file_open_ctx (void *ctx, notmuch_mailstore_t *mailstore, const char *filename);
/* Close a notmuch message previously opened with notmuch_message_open. */
void
@@ -406,7 +406,7 @@ char *
notmuch_sha1_of_string (const char *str);
char *
-notmuch_sha1_of_file (const char *filename);
+notmuch_sha1_of_file (FILE *file);
/* tags.c */
diff --git a/lib/notmuch.h b/lib/notmuch.h
index c554f6d..76ac6fc 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -742,8 +742,8 @@ notmuch_message_get_replies (notmuch_message_t *message);
/* Get a filename for the email corresponding to 'message'.
*
- * The returned filename is an absolute filename, (the initial
- * component will match notmuch_database_get_path() ).
+ * The returned filename is an relative filename of the message within
+ * the mail store.
*
* The returned string belongs to the message so should not be
* modified or freed by the caller (nor should it be referenced after
@@ -757,6 +757,18 @@ notmuch_message_get_replies (notmuch_message_t *message);
const char *
notmuch_message_get_filename (notmuch_message_t *message);
+/* Return file handle to read the content of the message.
+ *
+ * This is a helper function which determines message filename and
+ * calls notmuch_mailstore_open_file().
+ */
+FILE *
+notmuch_message_fopen (notmuch_message_t *message);
+
+/* Get a pointer to the mailstore where the message is stored */
+notmuch_mailstore_t *
+notmuch_message_get_mailstore (notmuch_message_t *message);
+
/* Message flags */
typedef enum _notmuch_message_flag {
NOTMUCH_MESSAGE_FLAG_MATCH,
diff --git a/lib/sha1.c b/lib/sha1.c
index cc48108..a8991b1 100644
--- a/lib/sha1.c
+++ b/lib/sha1.c
@@ -74,9 +74,8 @@ notmuch_sha1_of_string (const char *str)
* file not found, etc.), this function returns NULL.
*/
char *
-notmuch_sha1_of_file (const char *filename)
+notmuch_sha1_of_file (FILE *file)
{
- FILE *file;
#define BLOCK_SIZE 4096
unsigned char block[BLOCK_SIZE];
size_t bytes_read;
@@ -84,7 +83,6 @@ notmuch_sha1_of_file (const char *filename)
unsigned char digest[SHA1_DIGEST_SIZE];
char *result;
- file = fopen (filename, "r");
if (file == NULL)
return NULL;
@@ -108,8 +106,6 @@ notmuch_sha1_of_file (const char *filename)
result = _hex_of_sha1_digest (digest);
- fclose (file);
-
return result;
}
diff --git a/notmuch-client.h b/notmuch-client.h
index 4794b8e..f5106cd 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -120,7 +120,7 @@ char *
query_string_from_args (void *ctx, int argc, char *argv[]);
notmuch_status_t
-show_message_body (const char *filename,
+show_message_body (FILE *file,
void (*show_part) (GMimeObject *part, int *part_count));
char *
diff --git a/notmuch-reply.c b/notmuch-reply.c
index 0d6b7cf..a471137 100644
--- a/notmuch-reply.c
+++ b/notmuch-reply.c
@@ -285,6 +285,7 @@ add_recipients_from_message (GMimeMessage *reply,
static int
notmuch_reply_format_default(void *ctx, notmuch_config_t *config, notmuch_query_t *query)
{
+ FILE *file;
GMimeMessage *reply;
notmuch_messages_t *messages;
notmuch_message_t *message;
@@ -350,7 +351,14 @@ notmuch_reply_format_default(void *ctx, notmuch_config_t *config, notmuch_query_
notmuch_message_get_header (message, "date"),
notmuch_message_get_header (message, "from"));
- show_message_body (notmuch_message_get_filename (message), reply_part);
+ file = notmuch_message_fopen (message);
+ if (file) {
+ show_message_body (file, reply_part);
+ fclose (file);
+ } else
+ fprintf (stderr, "Error opening %s: %s\n",
+ notmuch_message_get_filename (message),
+ strerror (errno));
notmuch_message_destroy (message);
}
diff --git a/notmuch-show.c b/notmuch-show.c
index 26581c1..0fcaacf 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -336,6 +336,8 @@ format_part_json (GMimeObject *part, int *part_count)
static void
show_message (void *ctx, const show_format_t *format, notmuch_message_t *message, int indent)
{
+ FILE *file;
+
fputs (format->message_start, stdout);
if (format->message)
format->message(ctx, message, indent);
@@ -346,8 +348,16 @@ show_message (void *ctx, const show_format_t *format, notmuch_message_t *message
fputs (format->header_end, stdout);
fputs (format->body_start, stdout);
- if (format->part)
- show_message_body (notmuch_message_get_filename (message), format->part);
+ if (format->part) {
+ file = notmuch_message_fopen (message);
+ if (file) {
+ show_message_body (file, format->part);
+ fclose (file);
+ } else
+ fprintf (stderr, "Error opening %s: %s\n",
+ notmuch_message_get_filename (message),
+ strerror (errno));
+ }
fputs (format->body_end, stdout);
fputs (format->message_end, stdout);
diff --git a/show-message.c b/show-message.c
index 05ced9c..2df859a 100644
--- a/show-message.c
+++ b/show-message.c
@@ -60,23 +60,15 @@ show_message_part (GMimeObject *part, int *part_count,
}
notmuch_status_t
-show_message_body (const char *filename,
+show_message_body (FILE *file,
void (*show_part) (GMimeObject *part, int *part_count))
{
GMimeStream *stream = NULL;
GMimeParser *parser = NULL;
GMimeMessage *mime_message = NULL;
notmuch_status_t ret = NOTMUCH_STATUS_SUCCESS;
- FILE *file = NULL;
int part_count = 0;
- file = fopen (filename, "r");
- if (! file) {
- fprintf (stderr, "Error opening %s: %s\n", filename, strerror (errno));
- ret = NOTMUCH_STATUS_FILE_ERROR;
- goto DONE;
- }
-
stream = g_mime_stream_file_new (file);
g_mime_stream_file_set_owner (GMIME_STREAM_FILE (stream), FALSE);
@@ -87,7 +79,6 @@ show_message_body (const char *filename,
show_message_part (g_mime_message_get_mime_part (mime_message),
&part_count, show_part);
- DONE:
if (mime_message)
g_object_unref (mime_message);
@@ -97,8 +88,5 @@ show_message_body (const char *filename,
if (stream)
g_object_unref (stream);
- if (file)
- fclose (file);
-
return ret;
}
--
1.7.0.2
^ permalink raw reply related [flat|nested] 25+ messages in thread
* [PATCH v3 6/6] Add 'cat' subcommand
2010-03-27 20:42 ` Michal Sojka
` (4 preceding siblings ...)
2010-03-27 20:44 ` [PATCH v3 5/6] Access messages through mail store interface Michal Sojka
@ 2010-03-27 20:44 ` Michal Sojka
2010-03-27 20:49 ` Mailstore abstraction & maildir synchronization Michal Sojka
6 siblings, 0 replies; 25+ messages in thread
From: Michal Sojka @ 2010-03-27 20:44 UTC (permalink / raw)
To: notmuch
This command dumps raw message identified by filename to standard
output. It uses mail store interface to get the message from the right
place.
notmuch.el was modified to use this command to access the raw message
content (view/save attachments and view raw message).
Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz>
---
emacs/notmuch.el | 8 +++++-
notmuch-client.h | 3 ++
notmuch-show.c | 62 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
notmuch.c | 4 +++
4 files changed, 75 insertions(+), 2 deletions(-)
diff --git a/emacs/notmuch.el b/emacs/notmuch.el
index 117a365..38ba0e8 100644
--- a/emacs/notmuch.el
+++ b/emacs/notmuch.el
@@ -357,7 +357,11 @@ buffer."
(defun notmuch-show-view-raw-message ()
"View the raw email of the current message."
(interactive)
- (view-file (notmuch-show-get-filename)))
+ (let ((filename (notmuch-show-get-filename)))
+ (let ((buf (get-buffer-create (concat "*notmuch-raw-" filename "*"))))
+ (switch-to-buffer buf)
+ (save-excursion
+ (call-process notmuch-command nil t nil "cat" filename)))))
(defmacro with-current-notmuch-show-message (&rest body)
"Evaluate body with current buffer set to the text of current message"
@@ -365,7 +369,7 @@ buffer."
(let ((filename (notmuch-show-get-filename)))
(let ((buf (generate-new-buffer (concat "*notmuch-msg-" filename "*"))))
(with-current-buffer buf
- (insert-file-contents filename nil nil nil t)
+ (call-process notmuch-command nil t nil "cat" filename)
,@body)
(kill-buffer buf)))))
diff --git a/notmuch-client.h b/notmuch-client.h
index f5106cd..b9ddad2 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -107,6 +107,9 @@ notmuch_tag_command (void *ctx, int argc, char *argv[]);
int
notmuch_search_tags_command (void *ctx, int argc, char *argv[]);
+int
+notmuch_cat_command (void *ctx, int argc, char *argv[]);
+
const char *
notmuch_time_relative_date (const void *ctx, time_t then);
diff --git a/notmuch-show.c b/notmuch-show.c
index 0fcaacf..537d2e2 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -510,3 +510,65 @@ notmuch_show_command (void *ctx, unused (int argc), unused (char *argv[]))
return 0;
}
+
+int
+notmuch_cat_command (void *ctx, unused (int argc), unused (char *argv[]))
+{
+ notmuch_config_t *config;
+ notmuch_database_t *notmuch;
+ notmuch_mailstore_t *mailstore;
+ int i;
+ FILE *file;
+ const char *filename;
+ size_t size;
+ char buf[4096];
+
+ for (i = 0; i < argc && argv[i][0] == '-'; i++) {
+/* if (STRNCMP_LITERAL (argv[i], "--part=") == 0) { */
+/* part = atoi(argv[i] + sizeof ("--part=") - 1); */
+/* } else { */
+ fprintf (stderr, "Unrecognized option: %s\n", argv[i]);
+/* return 1; */
+/* } */
+ }
+
+ argc -= i;
+ argv += i;
+
+ if (argc == 0) {
+ fprintf (stderr, "Error: No filename given\n");
+ return 1;
+ }
+ filename = argv[0];
+
+ config = notmuch_config_open (ctx, NULL, NULL);
+ if (config == NULL)
+ return 1;
+
+ mailstore = notmuch_config_get_mailstore (config);
+
+ if (mailstore == NULL) {
+ fprintf (stderr, "Error: I have no mailstore\n");
+ return 1;
+ }
+
+ notmuch = notmuch_database_open (notmuch_config_get_database_path (config),
+ NOTMUCH_DATABASE_MODE_READ_ONLY,
+ mailstore);
+
+ file = notmuch_mailstore_open_file(mailstore, filename);
+ if (file == NULL) {
+ fprintf (stderr, "Error: Cannot open %s in %s: %s\n", filename,
+ notmuch_mailstore_get_type (mailstore), strerror (errno));
+ return 1;
+ }
+ while (!feof (file)) {
+ size = fread(buf, 1, sizeof(buf), file);
+ fwrite (buf, size, 1, stdout);
+ }
+ fclose (file);
+
+ notmuch_database_close (notmuch);
+
+ return 0;
+}
diff --git a/notmuch.c b/notmuch.c
index 95f057e..f8bb8f5 100644
--- a/notmuch.c
+++ b/notmuch.c
@@ -294,6 +294,10 @@ command_t commands[] = {
"\t\tcontain tags only from messages that match the search-term(s).\n"
"\n"
"\t\tIn both cases the list will be alphabetically sorted." },
+ { "cat", notmuch_cat_command,
+ "<path>",
+ "\t\tDumps raw message identified by <path> to standard output.",
+ "\n" },
{ "help", notmuch_help_command,
"[<command>]",
"\t\tThis message, or more detailed help for the named command.",
--
1.7.0.2
^ permalink raw reply related [flat|nested] 25+ messages in thread
* Re: Mailstore abstraction & maildir synchronization
2010-03-27 20:42 ` Michal Sojka
` (5 preceding siblings ...)
2010-03-27 20:44 ` [PATCH v3 6/6] Add 'cat' subcommand Michal Sojka
@ 2010-03-27 20:49 ` Michal Sojka
6 siblings, 0 replies; 25+ messages in thread
From: Michal Sojka @ 2010-03-27 20:49 UTC (permalink / raw)
To: Stewart Smith, notmuch
On Sat, 27 Mar 2010, Michal Sojka wrote:
> On Fri, 26 Mar 2010, Michal Sojka wrote:
> > On Wed, 24 Mar 2010, Stewart Smith wrote:
> > > On Thu, 18 Mar 2010 16:39:36 +0100, Michal Sojka <sojkam1@fel.cvut.cz> wrote:
> > > > - Only file-based storage is suported. Notmuch access the files
> > > > directly, and not via the mailstore interface.
> > >
> > > It'll be great when this is fixed... should be trivial to add a git
> > > backend then.
> >
> > Yes, it seems to be quite trivial. I'll probably look at this tomorrow.
>
> Here it is. It was not so trivial, because it was needed to change
> absolute paths to relative ones at several places.
I forgot to mention that this is also available in
git://rtime.felk.cvut.cz/notmuch.git branch mailstore-abstraction-v3 and
can be viewed online at http://rtime.felk.cvut.cz/gitweb/notmuch.git.
Michal
^ permalink raw reply [flat|nested] 25+ messages in thread
end of thread, other threads:[~2010-03-27 20:49 UTC | newest]
Thread overview: 25+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-03-18 15:39 Mailstore abstraction & maildir synchronization Michal Sojka
2010-03-18 15:39 ` [PATCH 1/4] Mailstore abstraction interface Michal Sojka
2010-03-18 15:39 ` [PATCH 2/4] Convert mailstore abstraction Michal Sojka
2010-03-18 15:39 ` [PATCH 3/4] Add maildir-based mailstore Michal Sojka
2010-03-23 10:56 ` Ruben Pollan
2010-03-23 13:35 ` Michal Sojka
2010-03-23 14:37 ` Ruben Pollan
2010-03-26 21:41 ` Michal Sojka
2010-03-26 21:42 ` [PATCH v2 1/4] Mailstore abstraction interface Michal Sojka
2010-03-26 21:42 ` [PATCH v2 2/4] Conversion to mailstore abstraction Michal Sojka
2010-03-26 21:42 ` [PATCH v2 3/4] Add maildir-based mailstore Michal Sojka
2010-03-26 21:42 ` [PATCH v2 4/4] Tests for " Michal Sojka
2010-03-18 15:39 ` [PATCH " Michal Sojka
2010-03-23 10:59 ` Ruben Pollan
2010-03-23 13:47 ` Michal Sojka
2010-03-24 5:06 ` Mailstore abstraction & maildir synchronization Stewart Smith
2010-03-26 22:08 ` Michal Sojka
2010-03-27 20:42 ` Michal Sojka
2010-03-27 20:44 ` [PATCH v3 1/6] Mailstore abstraction interface Michal Sojka
2010-03-27 20:44 ` [PATCH v3 2/6] Conversion to mailstore abstraction Michal Sojka
2010-03-27 20:44 ` [PATCH v3 3/6] Add maildir-based mailstore Michal Sojka
2010-03-27 20:44 ` [PATCH v3 4/6] Tests for " Michal Sojka
2010-03-27 20:44 ` [PATCH v3 5/6] Access messages through mail store interface Michal Sojka
2010-03-27 20:44 ` [PATCH v3 6/6] Add 'cat' subcommand Michal Sojka
2010-03-27 20:49 ` Mailstore abstraction & maildir synchronization Michal Sojka
Code repositories for project(s) associated with this public inbox
https://yhetil.org/notmuch.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).