[PATCH 0/3] Configuration file option to exclude files/directories

[PATCH 0/3] Configuration file option to exclude files/directories

[tomi.ollila]

This patch set adds a configuration option 'database.exclude'; a list of
files/directories to be excluded when doing 'notmuch new' operation.

Notes:

1) Currently the comments for newly created configuration file are not
updated, so for not this is 'undocumented feature'. Should there be an
empty configuration line as a placeholder ... ?

2) Whenever some already existing directory is added to the exclude list
and the parent directory timestamp has not changed, notmuch new will not
notice the directory has gone (as it still is there), user needs to 'touch'
the parent directory before next 'notmuch new' no make notmuch notice.

3) count_files() function is not touched. The functionality there has fallen
behind of add_files_recursive (maildir+tmp check and following symlinks).
The question there should it be updated, or attempted to merge with
add_files (as the comment says). count_files() is only called at the beginning
when database is not yet initialised.

Tomi

 notmuch-client.h |    3 ++
 notmuch-config.c |   89 +++++++++++++++++++++++++++++++++++++++--------------
 notmuch-new.c    |   22 ++++++++++++-
 3 files changed, 88 insertions(+), 26 deletions(-)

[PATCH 1/3] added function notmuch_talloc_g_key_file_get_string_list()

[tomi.ollila]

From: Tomi Ollila <tomi.ollila@iki.fi>

The function notmuch_talloc_g_key_file_get_string_list() wraps
call to g_key_file_get_string_list() in a way that the returned
string array is copied to talloc'd memory area. The returned
pointer is itself also a talloc context, child of the context
given as first argument.
---
 notmuch-config.c |   44 +++++++++++++++++++++++++++++++++++++++++---
 1 files changed, 41 insertions(+), 3 deletions(-)

diff --git a/notmuch-config.c b/notmuch-config.c
index 485fa72..706f481 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -170,6 +170,44 @@ get_username_from_passwd_file (void *ctx)
     return name;
 }
 
+/** XXX move to (not-yet-existent) notmuch-talloc.c, or somewhere */
+static char **
+notmuch_talloc_g_key_file_get_string_list (const void * ctx,
+					   GKeyFile *key_file,
+					   const gchar *group_name,
+					   const gchar *key,
+					   gsize *length,
+					   GError **error)
+{
+    char ** newlist;
+    gchar ** strlist = g_key_file_get_string_list (key_file, group_name, key,
+						   length, error);
+    if (strlist) {
+	int i;
+	int l = *length;
+
+	newlist = talloc_array (ctx, char *, l + 1);
+	if (newlist == NULL)
+	    goto fail1;
+	for (i = 0; i < l; i++) {
+	    if ( (newlist[i] = talloc_strdup (newlist, strlist[i])) == NULL)
+		goto fail2;
+	}
+	newlist[i] = NULL;
+	g_strfreev (strlist);
+
+	return newlist;
+    }
+    return NULL;
+
+fail2:
+    talloc_free (newlist);
+fail1:
+    g_strfreev (strlist);
+    *length = 0; /* like in g_key_file_get_string_list () */
+    return NULL;
+}
+
 /* Open the named notmuch configuration file. If the filename is NULL,
  * the value of the environment variable $NOTMUCH_CONFIG will be used.
  * If $NOTMUCH_CONFIG is unset, the default configuration file
@@ -229,7 +267,7 @@ notmuch_config_open (void *ctx,
 	fprintf (stderr, "Out of memory.\n");
 	return NULL;
     }
-    
+
     talloc_set_destructor (config, notmuch_config_destructor);
 
     if (filename) {
@@ -393,7 +431,7 @@ notmuch_config_open (void *ctx,
 }
 
 /* Close the given notmuch_config_t object, freeing all resources.
- * 
+ *
  * Note: Any changes made to the configuration are *not* saved by this
  * function. To save changes, call notmuch_config_save before
  * notmuch_config_close.
@@ -653,7 +691,7 @@ notmuch_config_command_get (void *ctx, char *item)
     } else if (strcmp(item, "user.other_email") == 0) {
 	const char **other_email;
 	size_t i, length;
-	
+
 	other_email = notmuch_config_get_user_other_email (config, &length);
 	for (i = 0; i < length; i++)
 	    printf ("%s\n", other_email[i]);
-- 
1.7.3.4

[PATCH 2/3] Made notmuch_config_get_new_tags() use notmuch_talloc_g_key_file_get_string_list().

[tomi.ollila]

From: Tomi Ollila <tomi.ollila@iki.fi>

---
 notmuch-config.c |   26 +++++---------------------
 1 files changed, 5 insertions(+), 21 deletions(-)

diff --git a/notmuch-config.c b/notmuch-config.c
index 706f481..648639b 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -606,29 +606,13 @@ const char **
 notmuch_config_get_new_tags (notmuch_config_t *config,
 			     size_t *length)
 {
-    char **tags;
-    size_t tags_length;
-    unsigned int i;
-
     if (config->new_tags == NULL) {
-	tags = g_key_file_get_string_list (config->key_file,
-					   "new", "tags",
-					   &tags_length, NULL);
-	if (tags) {
-	    config->new_tags = talloc_size (config,
-					    sizeof (char *) *
-					    (tags_length + 1));
-	    for (i = 0; i < tags_length; i++)
-		config->new_tags[i] = talloc_strdup (config->new_tags,
-						     tags[i]);
-	    config->new_tags[i] = NULL;
-
-	    g_strfreev (tags);
-
-	    config->new_tags_length = tags_length;
-	}
+	config->new_tags = notmuch_talloc_g_key_file_get_string_list(
+	    config,
+	    config->key_file,
+	    "new", "tags",
+	    &config->new_tags_length, NULL);
     }
-
     *length = config->new_tags_length;
     return config->new_tags;
 }
-- 
1.7.3.4

[PATCH 3/3] added support for user-specified directories to exclude

[tomi.ollila]

From: Tomi Ollila <tomi.ollila@iki.fi>

A new configuration key 'database.exclude' is used to determine
which directories user wants not to be scanned for new mails.
---
 notmuch-client.h |    3 +++
 notmuch-config.c |   19 +++++++++++++++++++
 notmuch-new.c    |   22 ++++++++++++++++++++--
 3 files changed, 42 insertions(+), 2 deletions(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index b50cb38..503ac79 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -197,6 +197,9 @@ void
 notmuch_config_set_database_path (notmuch_config_t *config,
 				  const char *database_path);
 
+const char **
+notmuch_config_get_database_exclude (notmuch_config_t *config,
+				     size_t *length);
 const char *
 notmuch_config_get_user_name (notmuch_config_t *config);
 
diff --git a/notmuch-config.c b/notmuch-config.c
index 648639b..b628074 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -88,6 +88,8 @@ struct _notmuch_config {
     GKeyFile *key_file;
 
     char *database_path;
+    const char **database_exclude;
+    size_t database_exclude_length;
     char *user_name;
     char *user_primary_email;
     const char **user_other_email;
@@ -282,6 +284,8 @@ notmuch_config_open (void *ctx,
     config->key_file = g_key_file_new ();
 
     config->database_path = NULL;
+    config->database_exclude = NULL;
+    config->database_exclude_length = 0;
     config->user_name = NULL;
     config->user_primary_email = NULL;
     config->user_other_email = NULL;
@@ -502,6 +506,21 @@ notmuch_config_set_database_path (notmuch_config_t *config,
     config->database_path = NULL;
 }
 
+const char **
+notmuch_config_get_database_exclude (notmuch_config_t *config,
+				     size_t *length)
+{
+    if (config->database_exclude == NULL) {
+	config->database_exclude = notmuch_talloc_g_key_file_get_string_list(
+	    config,
+	    config->key_file,
+	    "database", "exclude",
+	    &config->database_exclude_length, NULL);
+    }
+    *length = config->database_exclude_length;
+    return config->database_exclude;
+}
+
 const char *
 notmuch_config_get_user_name (notmuch_config_t *config)
 {
diff --git a/notmuch-new.c b/notmuch-new.c
index 7d17793..36da15f 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -38,6 +38,8 @@ typedef struct {
     int verbose;
     const char **new_tags;
     size_t new_tags_length;
+    const char **database_exclude;
+    size_t database_exclude_length;
 
     int total_files;
     int processed_files;
@@ -293,6 +295,8 @@ add_files_recursive (notmuch_database_t *notmuch,
     is_maildir = _entries_resemble_maildir (fs_entries, num_fs_entries);
 
     for (i = 0; i < num_fs_entries; i++) {
+	size_t j;
+
 	if (interrupted)
 	    break;
 
@@ -316,8 +320,6 @@ add_files_recursive (notmuch_database_t *notmuch,
 	 * Also ignore the .notmuch directory and any "tmp" directory
 	 * that appears within a maildir.
 	 */
-	/* XXX: Eventually we'll want more sophistication to let the
-	 * user specify files to be ignored. */
 	if (strcmp (entry->d_name, ".") == 0 ||
 	    strcmp (entry->d_name, "..") == 0 ||
 	    (is_maildir && strcmp (entry->d_name, "tmp") == 0) ||
@@ -325,6 +327,12 @@ add_files_recursive (notmuch_database_t *notmuch,
 	{
 	    continue;
 	}
+	/* Ignore user-specified directories */
+	for (j = 0; j < state->database_exclude_length; j++)
+	    if (strcmp(entry->d_name, state->database_exclude[j]) == 0)
+		break;
+	if (j < state->database_exclude_length)
+	    continue;
 
 	next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
 	status = add_files_recursive (notmuch, next, state);
@@ -357,11 +365,20 @@ add_files_recursive (notmuch_database_t *notmuch,
     /* Pass 2: Scan for new files, removed files, and removed directories. */
     for (i = 0; i < num_fs_entries; i++)
     {
+	size_t j;
+
 	if (interrupted)
 	    break;
 
         entry = fs_entries[i];
 
+	/* Ignore user-specified files & directories */
+	for (j = 0; j < state->database_exclude_length; j++)
+	    if (strcmp(entry->d_name, state->database_exclude[j]) == 0)
+		break;
+	if (j < state->database_exclude_length)
+	    continue;
+
 	/* Check if we've walked past any names in db_files or
 	 * db_subdirs. If so, these have been deleted. */
 	while (notmuch_filenames_valid (db_files) &&
@@ -800,6 +817,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
 	return 1;
 
     add_files_state.new_tags = notmuch_config_get_new_tags (config, &add_files_state.new_tags_length);
+    add_files_state.database_exclude = notmuch_config_get_database_exclude (config, &add_files_state.database_exclude_length);
     add_files_state.synchronize_flags = notmuch_config_get_maildir_synchronize_flags (config);
     add_files_state.message_ids_to_sync = _filename_list_create (ctx);
     db_path = notmuch_config_get_database_path (config);
-- 
1.7.3.4

Re: [PATCH 0/3] Configuration file option to exclude files/directories

[David Bremner]

On Wed, 14 Sep 2011 00:32:01 +0300, tomi.ollila@iki.fi wrote:
> This patch set adds a configuration option 'database.exclude'; a list of
> files/directories to be excluded when doing 'notmuch new' operation.

Hi All;

I could see idea of excluding spam and trash folders from indexing being
useful, but this has been sitting in the list for a few months without
comment. Does that mean no-one but Tomi thinks the functionality is
interesting? Or it just slipped by?

d

Re: [PATCH 0/3] Configuration file option to exclude files/directories

[Tom Prince]

On Wed, 07 Dec 2011 19:51:32 -0400, David Bremner <david@tethera.net> wrote:
> I could see idea of excluding spam and trash folders from indexing being
> useful, but this has been sitting in the list for a few months without
> comment. Does that mean no-one but Tomi thinks the functionality is
> interesting? Or it just slipped by?

A while a go, I had hacked the code to add .git to the list of excluded
dirctories, and I have since just moved .git under .notmuch. I think
this ability would be useful.

  Tom

Re: [PATCH 0/3] Configuration file option to exclude files/directories

[Tomi Ollila]

On Wed, 07 Dec 2011 19:51:32 -0400, David Bremner <david@tethera.net> wrote:
> On Wed, 14 Sep 2011 00:32:01 +0300, tomi.ollila@iki.fi wrote:
> > This patch set adds a configuration option 'database.exclude'; a list of
> > files/directories to be excluded when doing 'notmuch new' operation.
> 
> Hi All;
> 
> I could see idea of excluding spam and trash folders from indexing being
> useful, but this has been sitting in the list for a few months without
> comment. Does that mean no-one but Tomi thinks the functionality is
> interesting? Or it just slipped by?

Well, basically due to my comments in 
id:"1315949524-4948-1-git-send-email-tomi.ollila@iki.fi" (and so far little
interest to this patch series) I have 'drafts' and 'spam' directories
elsewhere. 

If there is renewed interests with real use cases I'm interested to make
this work more complete (now that there is change to get patches in ;) 
-- and then move my drafts and spam under database.path too.

I'll mark the current patches as 'notmuch::wip' for the time being.

> 
> d
> 

Tomi

Re: [PATCH 0/3] Configuration file option to exclude files/directories

[Pieter Praet]

On Thu, 08 Dec 2011 11:05:50 +0200, Tomi Ollila <tomi.ollila@iki.fi> wrote:
> On Wed, 07 Dec 2011 19:51:32 -0400, David Bremner <david@tethera.net> wrote:
> > On Wed, 14 Sep 2011 00:32:01 +0300, tomi.ollila@iki.fi wrote:
> > > This patch set adds a configuration option 'database.exclude'; a list of
> > > files/directories to be excluded when doing 'notmuch new' operation.
> > 
> > Hi All;
> > 
> > I could see idea of excluding spam and trash folders from indexing being
> > useful, but this has been sitting in the list for a few months without
> > comment. Does that mean no-one but Tomi thinks the functionality is
> > interesting? Or it just slipped by?
> 
> Well, basically due to my comments in 
> id:"1315949524-4948-1-git-send-email-tomi.ollila@iki.fi" (and so far little
> interest to this patch series) I have 'drafts' and 'spam' directories
> elsewhere. 
> 
> If there is renewed interests with real use cases I'm interested to make
> this work more complete (now that there is change to get patches in ;) 
> -- and then move my drafts and spam under database.path too.
> 

Please do!  The desire to be able to exclude certain files/folders has been
expressed many times in the past [1], and once again a mere 2 days ago [2].

I'm currently kicking myself for somehow having missed this thread...

Many thanks in advance!


> I'll mark the current patches as 'notmuch::wip' for the time being.
> 
> > 
> > d
> > 
> 
> Tomi
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch


Peace

-- 
Pieter

[1] id:"AANLkTikddX=fYiDmnN2Jx3ERcJndw=80+5_07=M45MQ=@mail.gmail.com"
[2] id:"20120122113212.GA7084@X200"

Re: [PATCH 0/3] Configuration file option to exclude files/directories

[David Edmondson]

[-- Attachment #1: Type: text/plain, Size: 1220 bytes --]

On Thu, 08 Dec 2011 11:05:50 +0200, Tomi Ollila <tomi.ollila@iki.fi> wrote:
> On Wed, 07 Dec 2011 19:51:32 -0400, David Bremner <david@tethera.net> wrote:
> > On Wed, 14 Sep 2011 00:32:01 +0300, tomi.ollila@iki.fi wrote:
> > > This patch set adds a configuration option 'database.exclude'; a list of
> > > files/directories to be excluded when doing 'notmuch new' operation.
> > 
> > Hi All;
> > 
> > I could see idea of excluding spam and trash folders from indexing being
> > useful, but this has been sitting in the list for a few months without
> > comment. Does that mean no-one but Tomi thinks the functionality is
> > interesting? Or it just slipped by?
> 
> Well, basically due to my comments in 
> id:"1315949524-4948-1-git-send-email-tomi.ollila@iki.fi" (and so far little
> interest to this patch series) I have 'drafts' and 'spam' directories
> elsewhere. 
> 
> If there is renewed interests with real use cases I'm interested to make
> this work more complete (now that there is change to get patches in ;) 
> -- and then move my drafts and spam under database.path too.

I'd be interested in these changes (I'd like to ignore .git). It would
help if a test case or two was include.

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

[PATCH 1/2] moved _notmuch_get_list () and _notmuch_set_list () up in file

[Tomi Ollila]

Moved _notmuch_get_list () and _notmuch_set_list () to a location
in notmuch-config.c so that new functions that will be located 
before the old location of those functions can also use these.
---

This patch is independent of the next one (just required by it)
and can (should) be pushed early.

 notmuch-config.c |   84 +++++++++++++++++++++++++++---------------------------
 1 files changed, 42 insertions(+), 42 deletions(-)

diff --git a/notmuch-config.c b/notmuch-config.c
index 0ded6d7..a124e34 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -467,6 +467,48 @@ notmuch_config_save (notmuch_config_t *config)
     return 0;
 }
 
+static const char **
+_config_get_list (notmuch_config_t *config,
+		  const char *section, const char *key,
+		  const char ***outlist, size_t *list_length, size_t *ret_length)
+{
+    assert(outlist);
+
+    if (*outlist == NULL) {
+
+	char **inlist = g_key_file_get_string_list (config->key_file,
+					     section, key, list_length, NULL);
+	if (inlist) {
+	    unsigned int i;
+
+	    *outlist = talloc_size (config, sizeof (char *) * (*list_length + 1));
+
+	    for (i = 0; i < *list_length; i++)
+		(*outlist)[i] = talloc_strdup (*outlist, inlist[i]);
+
+	    (*outlist)[i] = NULL;
+
+	    g_strfreev (inlist);
+	}
+    }
+
+    if (ret_length)
+	*ret_length = *list_length;
+
+    return *outlist;
+}
+
+static void
+_config_set_list (notmuch_config_t *config,
+		  const char *group, const char *name,
+		  const char *list[],
+		  size_t length, const char ***config_var )
+{
+    g_key_file_set_string_list (config->key_file, group, name, list, length);
+    talloc_free (*config_var);
+    *config_var = NULL;
+}
+
 const char *
 notmuch_config_get_database_path (notmuch_config_t *config)
 {
@@ -551,37 +593,6 @@ notmuch_config_set_user_primary_email (notmuch_config_t *config,
     config->user_primary_email = NULL;
 }
 
-static const char **
-_config_get_list (notmuch_config_t *config,
-		  const char *section, const char *key,
-		  const char ***outlist, size_t *list_length, size_t *ret_length)
-{
-    assert(outlist);
-
-    if (*outlist == NULL) {
-
-	char **inlist = g_key_file_get_string_list (config->key_file,
-					     section, key, list_length, NULL);
-	if (inlist) {
-	    unsigned int i;
-
-	    *outlist = talloc_size (config, sizeof (char *) * (*list_length + 1));
-
-	    for (i = 0; i < *list_length; i++)
-		(*outlist)[i] = talloc_strdup (*outlist, inlist[i]);
-
-	    (*outlist)[i] = NULL;
-
-	    g_strfreev (inlist);
-	}
-    }
-
-    if (ret_length)
-	*ret_length = *list_length;
-
-    return *outlist;
-}
-
 const char **
 notmuch_config_get_user_other_email (notmuch_config_t *config,   size_t *length)
 {
@@ -598,17 +609,6 @@ notmuch_config_get_new_tags (notmuch_config_t *config,   size_t *length)
 			     &(config->new_tags_length), length);
 }
 
-static void
-_config_set_list (notmuch_config_t *config,
-		  const char *group, const char *name,
-		  const char *list[],
-		  size_t length, const char ***config_var )
-{
-    g_key_file_set_string_list (config->key_file, group, name, list, length);
-    talloc_free (*config_var);
-    *config_var = NULL;
-}
-
 void
 notmuch_config_set_user_other_email (notmuch_config_t *config,
 				     const char *list[],
-- 
1.7.6.4

[PATCH 2/2] added support for user-specified directories to exclude

[Tomi Ollila]

A new configuration key 'database.exclude' is used to determine
which directories user wants not to be scanned for new mails.

---

Notes (from 2011-09-13):

1) Currently the comments for newly created configuration file are not
updated, so for not this is 'undocumented feature'. Should there be an
empty configuration line as a placeholder ... ?

2) Whenever some already existing directory is added to the exclude list
and the parent directory timestamp has not changed, notmuch new will not
notice the directory has gone (as it still is there), user needs to 'touch'
the parent directory before next 'notmuch new' no make notmuch notice.

2012-01-26: could notmuch track mtime of the configuration file and if
that changes, ignore mail directory timestamps ?

3) count_files() function is not touched. The functionality there has fallen
behind of add_files_recursive (maildir+tmp check and following symlinks).
The question there should it be updated, or attempted to merge with
add_files (as the comment says). count_files() is only called at the beginning
when database is not yet initialised.
---
 notmuch-client.h |    3 +++
 notmuch-config.c |   13 +++++++++++++
 notmuch-new.c    |   22 ++++++++++++++++++++--
 3 files changed, 36 insertions(+), 2 deletions(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index e0eb594..78460fc 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -219,6 +219,9 @@ void
 notmuch_config_set_database_path (notmuch_config_t *config,
 				  const char *database_path);
 
+const char **
+notmuch_config_get_database_exclude (notmuch_config_t *config,
+				     size_t *length);
 const char *
 notmuch_config_get_user_name (notmuch_config_t *config);
 
diff --git a/notmuch-config.c b/notmuch-config.c
index a124e34..e236114 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -99,6 +99,8 @@ struct _notmuch_config {
     GKeyFile *key_file;
 
     char *database_path;
+    const char **database_exclude;
+    size_t database_exclude_length;
     char *user_name;
     char *user_primary_email;
     const char **user_other_email;
@@ -258,6 +260,8 @@ notmuch_config_open (void *ctx,
     config->key_file = g_key_file_new ();
 
     config->database_path = NULL;
+    config->database_exclude = NULL;
+    config->database_exclude_length = 0;
     config->user_name = NULL;
     config->user_primary_email = NULL;
     config->user_other_email = NULL;
@@ -537,6 +541,15 @@ notmuch_config_set_database_path (notmuch_config_t *config,
     config->database_path = NULL;
 }
 
+const char **
+notmuch_config_get_database_exclude (notmuch_config_t *config,
+				     size_t *length)
+{
+    return _config_get_list (config, "database", "exclude",
+			     &(config->database_exclude),
+			     &(config->database_exclude_length), length);
+}
+
 const char *
 notmuch_config_get_user_name (notmuch_config_t *config)
 {
diff --git a/notmuch-new.c b/notmuch-new.c
index a569a54..d607f5b 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -39,6 +39,8 @@ typedef struct {
     int verbose;
     const char **new_tags;
     size_t new_tags_length;
+    const char **database_exclude;
+    size_t database_exclude_length;
 
     int total_files;
     int processed_files;
@@ -300,6 +302,8 @@ add_files_recursive (notmuch_database_t *notmuch,
     is_maildir = _entries_resemble_maildir (fs_entries, num_fs_entries);
 
     for (i = 0; i < num_fs_entries; i++) {
+	size_t j;
+
 	if (interrupted)
 	    break;
 
@@ -323,8 +327,6 @@ add_files_recursive (notmuch_database_t *notmuch,
 	 * Also ignore the .notmuch directory and any "tmp" directory
 	 * that appears within a maildir.
 	 */
-	/* XXX: Eventually we'll want more sophistication to let the
-	 * user specify files to be ignored. */
 	if (strcmp (entry->d_name, ".") == 0 ||
 	    strcmp (entry->d_name, "..") == 0 ||
 	    (is_maildir && strcmp (entry->d_name, "tmp") == 0) ||
@@ -332,6 +334,12 @@ add_files_recursive (notmuch_database_t *notmuch,
 	{
 	    continue;
 	}
+	/* Ignore user-specified directories */
+	for (j = 0; j < state->database_exclude_length; j++)
+	    if (strcmp(entry->d_name, state->database_exclude[j]) == 0)
+		break;
+	if (j < state->database_exclude_length)
+	    continue;
 
 	next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
 	status = add_files_recursive (notmuch, next, state);
@@ -364,11 +372,20 @@ add_files_recursive (notmuch_database_t *notmuch,
     /* Pass 2: Scan for new files, removed files, and removed directories. */
     for (i = 0; i < num_fs_entries; i++)
     {
+	size_t j;
+
 	if (interrupted)
 	    break;
 
         entry = fs_entries[i];
 
+	/* Ignore user-specified files & directories */
+	for (j = 0; j < state->database_exclude_length; j++)
+	    if (strcmp(entry->d_name, state->database_exclude[j]) == 0)
+		break;
+	if (j < state->database_exclude_length)
+	    continue;
+
 	/* Check if we've walked past any names in db_files or
 	 * db_subdirs. If so, these have been deleted. */
 	while (notmuch_filenames_valid (db_files) &&
@@ -837,6 +854,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
 	return 1;
 
     add_files_state.new_tags = notmuch_config_get_new_tags (config, &add_files_state.new_tags_length);
+    add_files_state.database_exclude = notmuch_config_get_database_exclude (config, &add_files_state.database_exclude_length);
     add_files_state.synchronize_flags = notmuch_config_get_maildir_synchronize_flags (config);
     db_path = notmuch_config_get_database_path (config);
 
-- 
1.7.6.4

Re: [PATCH 1/2] moved _notmuch_get_list () and _notmuch_set_list () up in file

[Jani Nikula]

On Thu, 26 Jan 2012 12:11:57 +0200, Tomi Ollila <tomi.ollila@iki.fi> wrote:
> Moved _notmuch_get_list () and _notmuch_set_list () to a location
> in notmuch-config.c so that new functions that will be located 
> before the old location of those functions can also use these.

Parse error. ;)

You mean something along the lines of: "Move _notmuch_get_list () and
_notmuch_set_list () earlier in the file to avoid forward declarations
in further work. No functional changes."

I'm sure native speakers can bikeshed that further. ;)

BR,
Jani.


> ---
> 
> This patch is independent of the next one (just required by it)
> and can (should) be pushed early.
> 
>  notmuch-config.c |   84 +++++++++++++++++++++++++++---------------------------
>  1 files changed, 42 insertions(+), 42 deletions(-)
> 
> diff --git a/notmuch-config.c b/notmuch-config.c
> index 0ded6d7..a124e34 100644
> --- a/notmuch-config.c
> +++ b/notmuch-config.c
> @@ -467,6 +467,48 @@ notmuch_config_save (notmuch_config_t *config)
>      return 0;
>  }
>  
> +static const char **
> +_config_get_list (notmuch_config_t *config,
> +		  const char *section, const char *key,
> +		  const char ***outlist, size_t *list_length, size_t *ret_length)
> +{
> +    assert(outlist);
> +
> +    if (*outlist == NULL) {
> +
> +	char **inlist = g_key_file_get_string_list (config->key_file,
> +					     section, key, list_length, NULL);
> +	if (inlist) {
> +	    unsigned int i;
> +
> +	    *outlist = talloc_size (config, sizeof (char *) * (*list_length + 1));
> +
> +	    for (i = 0; i < *list_length; i++)
> +		(*outlist)[i] = talloc_strdup (*outlist, inlist[i]);
> +
> +	    (*outlist)[i] = NULL;
> +
> +	    g_strfreev (inlist);
> +	}
> +    }
> +
> +    if (ret_length)
> +	*ret_length = *list_length;
> +
> +    return *outlist;
> +}
> +
> +static void
> +_config_set_list (notmuch_config_t *config,
> +		  const char *group, const char *name,
> +		  const char *list[],
> +		  size_t length, const char ***config_var )
> +{
> +    g_key_file_set_string_list (config->key_file, group, name, list, length);
> +    talloc_free (*config_var);
> +    *config_var = NULL;
> +}
> +
>  const char *
>  notmuch_config_get_database_path (notmuch_config_t *config)
>  {
> @@ -551,37 +593,6 @@ notmuch_config_set_user_primary_email (notmuch_config_t *config,
>      config->user_primary_email = NULL;
>  }
>  
> -static const char **
> -_config_get_list (notmuch_config_t *config,
> -		  const char *section, const char *key,
> -		  const char ***outlist, size_t *list_length, size_t *ret_length)
> -{
> -    assert(outlist);
> -
> -    if (*outlist == NULL) {
> -
> -	char **inlist = g_key_file_get_string_list (config->key_file,
> -					     section, key, list_length, NULL);
> -	if (inlist) {
> -	    unsigned int i;
> -
> -	    *outlist = talloc_size (config, sizeof (char *) * (*list_length + 1));
> -
> -	    for (i = 0; i < *list_length; i++)
> -		(*outlist)[i] = talloc_strdup (*outlist, inlist[i]);
> -
> -	    (*outlist)[i] = NULL;
> -
> -	    g_strfreev (inlist);
> -	}
> -    }
> -
> -    if (ret_length)
> -	*ret_length = *list_length;
> -
> -    return *outlist;
> -}
> -
>  const char **
>  notmuch_config_get_user_other_email (notmuch_config_t *config,   size_t *length)
>  {
> @@ -598,17 +609,6 @@ notmuch_config_get_new_tags (notmuch_config_t *config,   size_t *length)
>  			     &(config->new_tags_length), length);
>  }
>  
> -static void
> -_config_set_list (notmuch_config_t *config,
> -		  const char *group, const char *name,
> -		  const char *list[],
> -		  size_t length, const char ***config_var )
> -{
> -    g_key_file_set_string_list (config->key_file, group, name, list, length);
> -    talloc_free (*config_var);
> -    *config_var = NULL;
> -}
> -
>  void
>  notmuch_config_set_user_other_email (notmuch_config_t *config,
>  				     const char *list[],
> -- 
> 1.7.6.4
> 
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 2/2] added support for user-specified directories to exclude

[Jani Nikula]

On Thu, 26 Jan 2012 12:11:58 +0200, Tomi Ollila <tomi.ollila@iki.fi> wrote:
> A new configuration key 'database.exclude' is used to determine
> which directories user wants not to be scanned for new mails.
> 
> ---
> 
> Notes (from 2011-09-13):
> 
> 1) Currently the comments for newly created configuration file are not
> updated, so for not this is 'undocumented feature'. Should there be an
> empty configuration line as a placeholder ... ?
> 
> 2) Whenever some already existing directory is added to the exclude list
> and the parent directory timestamp has not changed, notmuch new will not
> notice the directory has gone (as it still is there), user needs to 'touch'
> the parent directory before next 'notmuch new' no make notmuch notice.
> 
> 2012-01-26: could notmuch track mtime of the configuration file and if
> that changes, ignore mail directory timestamps ?
> 
> 3) count_files() function is not touched. The functionality there has fallen
> behind of add_files_recursive (maildir+tmp check and following symlinks).
> The question there should it be updated, or attempted to merge with
> add_files (as the comment says). count_files() is only called at the beginning
> when database is not yet initialised.
> ---
>  notmuch-client.h |    3 +++
>  notmuch-config.c |   13 +++++++++++++
>  notmuch-new.c    |   22 ++++++++++++++++++++--
>  3 files changed, 36 insertions(+), 2 deletions(-)
> 
> diff --git a/notmuch-client.h b/notmuch-client.h
> index e0eb594..78460fc 100644
> --- a/notmuch-client.h
> +++ b/notmuch-client.h
> @@ -219,6 +219,9 @@ void
>  notmuch_config_set_database_path (notmuch_config_t *config,
>  				  const char *database_path);
>  
> +const char **
> +notmuch_config_get_database_exclude (notmuch_config_t *config,
> +				     size_t *length);
>  const char *
>  notmuch_config_get_user_name (notmuch_config_t *config);
>  
> diff --git a/notmuch-config.c b/notmuch-config.c
> index a124e34..e236114 100644
> --- a/notmuch-config.c
> +++ b/notmuch-config.c
> @@ -99,6 +99,8 @@ struct _notmuch_config {
>      GKeyFile *key_file;
>  
>      char *database_path;
> +    const char **database_exclude;
> +    size_t database_exclude_length;
>      char *user_name;
>      char *user_primary_email;
>      const char **user_other_email;
> @@ -258,6 +260,8 @@ notmuch_config_open (void *ctx,
>      config->key_file = g_key_file_new ();
>  
>      config->database_path = NULL;
> +    config->database_exclude = NULL;
> +    config->database_exclude_length = 0;
>      config->user_name = NULL;
>      config->user_primary_email = NULL;
>      config->user_other_email = NULL;
> @@ -537,6 +541,15 @@ notmuch_config_set_database_path (notmuch_config_t *config,
>      config->database_path = NULL;
>  }
>  
> +const char **
> +notmuch_config_get_database_exclude (notmuch_config_t *config,
> +				     size_t *length)
> +{
> +    return _config_get_list (config, "database", "exclude",
> +			     &(config->database_exclude),
> +			     &(config->database_exclude_length), length);
> +}
> +
>  const char *
>  notmuch_config_get_user_name (notmuch_config_t *config)
>  {
> diff --git a/notmuch-new.c b/notmuch-new.c
> index a569a54..d607f5b 100644
> --- a/notmuch-new.c
> +++ b/notmuch-new.c
> @@ -39,6 +39,8 @@ typedef struct {
>      int verbose;
>      const char **new_tags;
>      size_t new_tags_length;
> +    const char **database_exclude;
> +    size_t database_exclude_length;
>  
>      int total_files;
>      int processed_files;
> @@ -300,6 +302,8 @@ add_files_recursive (notmuch_database_t *notmuch,
>      is_maildir = _entries_resemble_maildir (fs_entries, num_fs_entries);
>  
>      for (i = 0; i < num_fs_entries; i++) {
> +	size_t j;
> +
>  	if (interrupted)
>  	    break;
>  
> @@ -323,8 +327,6 @@ add_files_recursive (notmuch_database_t *notmuch,
>  	 * Also ignore the .notmuch directory and any "tmp" directory
>  	 * that appears within a maildir.
>  	 */
> -	/* XXX: Eventually we'll want more sophistication to let the
> -	 * user specify files to be ignored. */
>  	if (strcmp (entry->d_name, ".") == 0 ||
>  	    strcmp (entry->d_name, "..") == 0 ||
>  	    (is_maildir && strcmp (entry->d_name, "tmp") == 0) ||
> @@ -332,6 +334,12 @@ add_files_recursive (notmuch_database_t *notmuch,
>  	{
>  	    continue;
>  	}
> +	/* Ignore user-specified directories */
> +	for (j = 0; j < state->database_exclude_length; j++)
> +	    if (strcmp(entry->d_name, state->database_exclude[j]) == 0)
> +		break;
> +	if (j < state->database_exclude_length)
> +	    continue;

How about wrapping that in a function you can use here and below?

	if (user_wants_this_excluded (...))
		continue;

Please also have a look at id:"87pqecylon.fsf@nikula.org" and the
patches Austin posted. "Auto ignore"?

BR,
Jani.

>  
>  	next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
>  	status = add_files_recursive (notmuch, next, state);
> @@ -364,11 +372,20 @@ add_files_recursive (notmuch_database_t *notmuch,
>      /* Pass 2: Scan for new files, removed files, and removed directories. */
>      for (i = 0; i < num_fs_entries; i++)
>      {
> +	size_t j;
> +
>  	if (interrupted)
>  	    break;
>  
>          entry = fs_entries[i];
>  
> +	/* Ignore user-specified files & directories */
> +	for (j = 0; j < state->database_exclude_length; j++)
> +	    if (strcmp(entry->d_name, state->database_exclude[j]) == 0)
> +		break;
> +	if (j < state->database_exclude_length)
> +	    continue;
> +
>  	/* Check if we've walked past any names in db_files or
>  	 * db_subdirs. If so, these have been deleted. */
>  	while (notmuch_filenames_valid (db_files) &&
> @@ -837,6 +854,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
>  	return 1;
>  
>      add_files_state.new_tags = notmuch_config_get_new_tags (config, &add_files_state.new_tags_length);
> +    add_files_state.database_exclude = notmuch_config_get_database_exclude (config, &add_files_state.database_exclude_length);
>      add_files_state.synchronize_flags = notmuch_config_get_maildir_synchronize_flags (config);
>      db_path = notmuch_config_get_database_path (config);
>  
> -- 
> 1.7.6.4
> 
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 2/2] added support for user-specified directories to exclude

[Tomi Ollila]

On Thu, 26 Jan 2012 13:11:36 +0000, Jani Nikula <jani@nikula.org> wrote:
> On Thu, 26 Jan 2012 12:11:58 +0200, Tomi Ollila <tomi.ollila@iki.fi> wrote:
> > A new configuration key 'database.exclude' is used to determine
> > which directories user wants not to be scanned for new mails.
> > 
> > ---
> > 
> > Notes (from 2011-09-13):
> > 
> > 1) Currently the comments for newly created configuration file are not
> > updated, so for not this is 'undocumented feature'. Should there be an
> > empty configuration line as a placeholder ... ?
> > 
> > 2) Whenever some already existing directory is added to the exclude list
> > and the parent directory timestamp has not changed, notmuch new will not
> > notice the directory has gone (as it still is there), user needs to 'touch'
> > the parent directory before next 'notmuch new' no make notmuch notice.
> > 
> > 2012-01-26: could notmuch track mtime of the configuration file and if
> > that changes, ignore mail directory timestamps ?
> > 
> > 3) count_files() function is not touched. The functionality there has fallen
> > behind of add_files_recursive (maildir+tmp check and following symlinks).
> > The question there should it be updated, or attempted to merge with
> > add_files (as the comment says). count_files() is only called at the beginning
> > when database is not yet initialised.
> > ---

[ ... ]

> > +	/* Ignore user-specified directories */
> > +	for (j = 0; j < state->database_exclude_length; j++)
> > +	    if (strcmp(entry->d_name, state->database_exclude[j]) == 0)
> > +		break;
> > +	if (j < state->database_exclude_length)
> > +	    continue;
> 
> How about wrapping that in a function you can use here and below?
> 
> 	if (user_wants_this_excluded (...))
> 		continue;

Good point....

> Please also have a look at id:"87pqecylon.fsf@nikula.org" and the
> patches Austin posted. "Auto ignore"?

I personally don't mind auto ignore -- I even like the system yells about
those 3 files I forgot to add to ignore list every now and then (when the
mtime of the directory those are located changes). This is also something
different -- I can start keeping my drafts and spam directories alongside
other mail directories (or folders is you want to use that term).

Thanks for that id: it refers thread starting from id:"ylp7hi23mw8.fsf@tyndall.ie" 
( http://notmuchmail.org/pipermail/notmuch/2010/003145.html )

and the patch there is somewhat more complete (although old rebase-wise). 
It's easy to combine works together, but more difficult is to choose best 
terminology: database.exclude vs. new.ignore (or something in between or 
totally different). Ideas anyone?

Tomi

Re: [PATCH 1/2] moved _notmuch_get_list () and _notmuch_set_list () up in file

[Tomi Ollila]

On Thu, 26 Jan 2012 13:03:46 +0000, Jani Nikula <jani@nikula.org> wrote:
> On Thu, 26 Jan 2012 12:11:57 +0200, Tomi Ollila <tomi.ollila@iki.fi> wrote:
> > Moved _notmuch_get_list () and _notmuch_set_list () to a location
> > in notmuch-config.c so that new functions that will be located 
> > before the old location of those functions can also use these.
> 
> Parse error. ;)
> 
> You mean something along the lines of: "Move _notmuch_get_list () and
> _notmuch_set_list () earlier in the file to avoid forward declarations
> in further work. No functional changes."
> 
> I'm sure native speakers can bikeshed that further. ;)

Ok, they haven't. I'n resubmit this alone with better commit
message -- I look the other after I have better time.

> 
> BR,
> Jani.
> 

Tomi

[PATCH] moved _notmuch_(get|set)_list () functions earlyer in the file

[Tomi Ollila]

Moved static functions _notmuch_get_list () and _notmuch_set_list ()
closer to the beginning of file so that their definition is known
(without adding forward declarations) in further work.

No functional changes.
---
 notmuch-config.c |   84 +++++++++++++++++++++++++++---------------------------
 1 files changed, 42 insertions(+), 42 deletions(-)

diff --git a/notmuch-config.c b/notmuch-config.c
index 0ded6d7..a124e34 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -467,6 +467,48 @@ notmuch_config_save (notmuch_config_t *config)
     return 0;
 }
 
+static const char **
+_config_get_list (notmuch_config_t *config,
+		  const char *section, const char *key,
+		  const char ***outlist, size_t *list_length, size_t *ret_length)
+{
+    assert(outlist);
+
+    if (*outlist == NULL) {
+
+	char **inlist = g_key_file_get_string_list (config->key_file,
+					     section, key, list_length, NULL);
+	if (inlist) {
+	    unsigned int i;
+
+	    *outlist = talloc_size (config, sizeof (char *) * (*list_length + 1));
+
+	    for (i = 0; i < *list_length; i++)
+		(*outlist)[i] = talloc_strdup (*outlist, inlist[i]);
+
+	    (*outlist)[i] = NULL;
+
+	    g_strfreev (inlist);
+	}
+    }
+
+    if (ret_length)
+	*ret_length = *list_length;
+
+    return *outlist;
+}
+
+static void
+_config_set_list (notmuch_config_t *config,
+		  const char *group, const char *name,
+		  const char *list[],
+		  size_t length, const char ***config_var )
+{
+    g_key_file_set_string_list (config->key_file, group, name, list, length);
+    talloc_free (*config_var);
+    *config_var = NULL;
+}
+
 const char *
 notmuch_config_get_database_path (notmuch_config_t *config)
 {
@@ -551,37 +593,6 @@ notmuch_config_set_user_primary_email (notmuch_config_t *config,
     config->user_primary_email = NULL;
 }
 
-static const char **
-_config_get_list (notmuch_config_t *config,
-		  const char *section, const char *key,
-		  const char ***outlist, size_t *list_length, size_t *ret_length)
-{
-    assert(outlist);
-
-    if (*outlist == NULL) {
-
-	char **inlist = g_key_file_get_string_list (config->key_file,
-					     section, key, list_length, NULL);
-	if (inlist) {
-	    unsigned int i;
-
-	    *outlist = talloc_size (config, sizeof (char *) * (*list_length + 1));
-
-	    for (i = 0; i < *list_length; i++)
-		(*outlist)[i] = talloc_strdup (*outlist, inlist[i]);
-
-	    (*outlist)[i] = NULL;
-
-	    g_strfreev (inlist);
-	}
-    }
-
-    if (ret_length)
-	*ret_length = *list_length;
-
-    return *outlist;
-}
-
 const char **
 notmuch_config_get_user_other_email (notmuch_config_t *config,   size_t *length)
 {
@@ -598,17 +609,6 @@ notmuch_config_get_new_tags (notmuch_config_t *config,   size_t *length)
 			     &(config->new_tags_length), length);
 }
 
-static void
-_config_set_list (notmuch_config_t *config,
-		  const char *group, const char *name,
-		  const char *list[],
-		  size_t length, const char ***config_var )
-{
-    g_key_file_set_string_list (config->key_file, group, name, list, length);
-    talloc_free (*config_var);
-    *config_var = NULL;
-}
-
 void
 notmuch_config_set_user_other_email (notmuch_config_t *config,
 				     const char *list[],
-- 
1.7.8.2

Re: [PATCH 1/2] moved _notmuch_get_list () and _notmuch_set_list () up in file

[Ethan Glasser-Camp]

On 01/27/2012 05:42 AM, Tomi Ollila wrote:
> On Thu, 26 Jan 2012 13:03:46 +0000, Jani Nikula<jani@nikula.org>  wrote:
>> On Thu, 26 Jan 2012 12:11:57 +0200, Tomi Ollila<tomi.ollila@iki.fi>  wrote:
>>> Moved _notmuch_get_list () and _notmuch_set_list () to a location
>>> in notmuch-config.c so that new functions that will be located
>>> before the old location of those functions can also use these.
>> Parse error. ;)
>>
>> You mean something along the lines of: "Move _notmuch_get_list () and
>> _notmuch_set_list () earlier in the file to avoid forward declarations
>> in further work. No functional changes."
>>
>> I'm sure native speakers can bikeshed that further. ;)
> Ok, they haven't. I'n resubmit this alone with better commit
> message -- I look the other after I have better time.

As a native speaker, your new version is acceptable but I found "in 
further work" a little odd. (Depending on what you meant, I'd say "in 
upcoming patches".) The thing I found most confusing about the comment 
is that the functions aren't called _notmuch_get_list or 
_notmuch_set_list (instead they are _config_get_list and _config_set_list.)

Ethan

Re: [PATCH 2/2] added support for user-specified directories to exclude

[Austin Clements]

Quoth Tomi Ollila on Jan 27 at 12:41 pm:
> On Thu, 26 Jan 2012 13:11:36 +0000, Jani Nikula <jani@nikula.org> wrote:
> > On Thu, 26 Jan 2012 12:11:58 +0200, Tomi Ollila <tomi.ollila@iki.fi> wrote:
> > > A new configuration key 'database.exclude' is used to determine
> > > which directories user wants not to be scanned for new mails.
> > > 
> > > ---
> > > 
> > > Notes (from 2011-09-13):
> > > 
> > > 1) Currently the comments for newly created configuration file are not
> > > updated, so for not this is 'undocumented feature'. Should there be an
> > > empty configuration line as a placeholder ... ?
> > > 
> > > 2) Whenever some already existing directory is added to the exclude list
> > > and the parent directory timestamp has not changed, notmuch new will not
> > > notice the directory has gone (as it still is there), user needs to 'touch'
> > > the parent directory before next 'notmuch new' no make notmuch notice.
> > > 
> > > 2012-01-26: could notmuch track mtime of the configuration file and if
> > > that changes, ignore mail directory timestamps ?
> > > 
> > > 3) count_files() function is not touched. The functionality there has fallen
> > > behind of add_files_recursive (maildir+tmp check and following symlinks).
> > > The question there should it be updated, or attempted to merge with
> > > add_files (as the comment says). count_files() is only called at the beginning
> > > when database is not yet initialised.
> > > ---
> 
> [ ... ]
> 
> > > +	/* Ignore user-specified directories */
> > > +	for (j = 0; j < state->database_exclude_length; j++)
> > > +	    if (strcmp(entry->d_name, state->database_exclude[j]) == 0)
> > > +		break;
> > > +	if (j < state->database_exclude_length)
> > > +	    continue;
> > 
> > How about wrapping that in a function you can use here and below?
> > 
> > 	if (user_wants_this_excluded (...))
> > 		continue;
> 
> Good point....
> 
> > Please also have a look at id:"87pqecylon.fsf@nikula.org" and the
> > patches Austin posted. "Auto ignore"?
> 
> I personally don't mind auto ignore -- I even like the system yells about
> those 3 files I forgot to add to ignore list every now and then (when the
> mtime of the directory those are located changes). This is also something
> different -- I can start keeping my drafts and spam directories alongside
> other mail directories (or folders is you want to use that term).

The allure of auto ignore is that you don't have to do anything to use
it, but it's much less powerful than explicit ignores.  I think having
both mechanisms would be overkill and I would use explicit ignores in
preference to auto ignore.

> Thanks for that id: it refers thread starting from id:"ylp7hi23mw8.fsf@tyndall.ie" 
> ( http://notmuchmail.org/pipermail/notmuch/2010/003145.html )
> 
> and the patch there is somewhat more complete (although old rebase-wise). 
> It's easy to combine works together, but more difficult is to choose best 
> terminology: database.exclude vs. new.ignore (or something in between or 
> totally different). Ideas anyone?

I would weigh in on the side of new.ignore, or at least something
under [new].  This option fits better with the existing option in
[new] than it does with the existing option in [database].

Re: [PATCH] moved _notmuch_(get|set)_list () functions earlyer in the file

[Austin Clements]

Quoth Tomi Ollila on Jan 27 at 12:59 pm:
> Moved static functions _notmuch_get_list () and _notmuch_set_list ()
> closer to the beginning of file so that their definition is known
> (without adding forward declarations) in further work.
> 
> No functional changes.

This patch LGTM.  This is where these functions should live
regardless.

Re: [PATCH 2/2] added support for user-specified directories to exclude

[Austin Clements]

Quoth Tomi Ollila on Jan 26 at 12:11 pm:
> A new configuration key 'database.exclude' is used to determine
> which directories user wants not to be scanned for new mails.

Not just directories.

> ---
> 
> Notes (from 2011-09-13):
> 
> 1) Currently the comments for newly created configuration file are not
> updated, so for not this is 'undocumented feature'. Should there be an
> empty configuration line as a placeholder ... ?

This is unfortunate and hard to do anything about with the current
config file design.  For precisely this reason, I think option
documentation should be in per-option comments instead of per-section
comments, but this would be a large change to the config structure.
For now, it should definitely be added to the section comment so that
newly generated config files get it.  If you want to be fancy, you can
check for the old section comment (or perhaps just its hash) and
automatically upgrade it.

> 2) Whenever some already existing directory is added to the exclude list
> and the parent directory timestamp has not changed, notmuch new will not
> notice the directory has gone (as it still is there), user needs to 'touch'
> the parent directory before next 'notmuch new' no make notmuch notice.
> 
> 2012-01-26: could notmuch track mtime of the configuration file and if
> that changes, ignore mail directory timestamps ?

notmuch actively descends into every subdirectory of your Maildir (it
has to, since mtime changes don't propagate up), which seems like the
perfect opportunity to check if an ignored directory is present in the
database.  This is harder for ignored files and probably not worth
solving since people are probably ignoring files that aren't mail
anyway (whereas I know people want to ignore directories that contain
mail).

> 3) count_files() function is not touched. The functionality there has fallen
> behind of add_files_recursive (maildir+tmp check and following symlinks).
> The question there should it be updated, or attempted to merge with
> add_files (as the comment says). count_files() is only called at the beginning
> when database is not yet initialised.

Given that count_files is already broken, it's probably best to leave
it as is for now and maybe fix it thoroughly later.  I wouldn't be too
worried about this though, since a new database is likely to also have
no configured ignores.

> ---
>  notmuch-client.h |    3 +++
>  notmuch-config.c |   13 +++++++++++++
>  notmuch-new.c    |   22 ++++++++++++++++++++--
>  3 files changed, 36 insertions(+), 2 deletions(-)
> 
> diff --git a/notmuch-client.h b/notmuch-client.h
> index e0eb594..78460fc 100644
> --- a/notmuch-client.h
> +++ b/notmuch-client.h
> @@ -219,6 +219,9 @@ void
>  notmuch_config_set_database_path (notmuch_config_t *config,
>  				  const char *database_path);
>  
> +const char **
> +notmuch_config_get_database_exclude (notmuch_config_t *config,
> +				     size_t *length);

As I mentioned in another email, I would prefer to see this in the
'new' section (though I could probably be persuaded otherwise).  Also,
I think "ignore" would be more typical terminology (think .gitignore,
etc).

Also, every other getter has a corresponding setter.

>  const char *
>  notmuch_config_get_user_name (notmuch_config_t *config);
>  
> diff --git a/notmuch-config.c b/notmuch-config.c
> index a124e34..e236114 100644
> --- a/notmuch-config.c
> +++ b/notmuch-config.c
> @@ -99,6 +99,8 @@ struct _notmuch_config {
>      GKeyFile *key_file;
>  
>      char *database_path;
> +    const char **database_exclude;
> +    size_t database_exclude_length;
>      char *user_name;
>      char *user_primary_email;
>      const char **user_other_email;
> @@ -258,6 +260,8 @@ notmuch_config_open (void *ctx,
>      config->key_file = g_key_file_new ();
>  
>      config->database_path = NULL;
> +    config->database_exclude = NULL;
> +    config->database_exclude_length = 0;
>      config->user_name = NULL;
>      config->user_primary_email = NULL;
>      config->user_other_email = NULL;
> @@ -537,6 +541,15 @@ notmuch_config_set_database_path (notmuch_config_t *config,
>      config->database_path = NULL;
>  }
>  
> +const char **
> +notmuch_config_get_database_exclude (notmuch_config_t *config,
> +				     size_t *length)
> +{
> +    return _config_get_list (config, "database", "exclude",
> +			     &(config->database_exclude),
> +			     &(config->database_exclude_length), length);
> +}
> +

All of the other options are fetched by notmuch_config_open and set if
they don't exist.  This has the effect that they will get added to the
configuration file if it is written out, which won't happen to this
option.

>  const char *
>  notmuch_config_get_user_name (notmuch_config_t *config)
>  {
> diff --git a/notmuch-new.c b/notmuch-new.c
> index a569a54..d607f5b 100644
> --- a/notmuch-new.c
> +++ b/notmuch-new.c
> @@ -39,6 +39,8 @@ typedef struct {
>      int verbose;
>      const char **new_tags;
>      size_t new_tags_length;
> +    const char **database_exclude;
> +    size_t database_exclude_length;
>  
>      int total_files;
>      int processed_files;
> @@ -300,6 +302,8 @@ add_files_recursive (notmuch_database_t *notmuch,
>      is_maildir = _entries_resemble_maildir (fs_entries, num_fs_entries);
>  
>      for (i = 0; i < num_fs_entries; i++) {
> +	size_t j;
> +
>  	if (interrupted)
>  	    break;
>  
> @@ -323,8 +327,6 @@ add_files_recursive (notmuch_database_t *notmuch,
>  	 * Also ignore the .notmuch directory and any "tmp" directory
>  	 * that appears within a maildir.
>  	 */
> -	/* XXX: Eventually we'll want more sophistication to let the
> -	 * user specify files to be ignored. */
>  	if (strcmp (entry->d_name, ".") == 0 ||
>  	    strcmp (entry->d_name, "..") == 0 ||
>  	    (is_maildir && strcmp (entry->d_name, "tmp") == 0) ||
> @@ -332,6 +334,12 @@ add_files_recursive (notmuch_database_t *notmuch,
>  	{
>  	    continue;
>  	}
> +	/* Ignore user-specified directories */
> +	for (j = 0; j < state->database_exclude_length; j++)
> +	    if (strcmp(entry->d_name, state->database_exclude[j]) == 0)
> +		break;
> +	if (j < state->database_exclude_length)
> +	    continue;

As Jani said, move this into its own function (which would simplify
the control flow, too).

>  
>  	next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
>  	status = add_files_recursive (notmuch, next, state);
> @@ -364,11 +372,20 @@ add_files_recursive (notmuch_database_t *notmuch,
>      /* Pass 2: Scan for new files, removed files, and removed directories. */
>      for (i = 0; i < num_fs_entries; i++)
>      {
> +	size_t j;
> +
>  	if (interrupted)
>  	    break;
>  
>          entry = fs_entries[i];
>  
> +	/* Ignore user-specified files & directories */
> +	for (j = 0; j < state->database_exclude_length; j++)
> +	    if (strcmp(entry->d_name, state->database_exclude[j]) == 0)
> +		break;
> +	if (j < state->database_exclude_length)
> +	    continue;
> +
>  	/* Check if we've walked past any names in db_files or
>  	 * db_subdirs. If so, these have been deleted. */
>  	while (notmuch_filenames_valid (db_files) &&
> @@ -837,6 +854,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
>  	return 1;
>  
>      add_files_state.new_tags = notmuch_config_get_new_tags (config, &add_files_state.new_tags_length);
> +    add_files_state.database_exclude = notmuch_config_get_database_exclude (config, &add_files_state.database_exclude_length);
>      add_files_state.synchronize_flags = notmuch_config_get_maildir_synchronize_flags (config);
>      db_path = notmuch_config_get_database_path (config);
>  

Re: [PATCH 2/2] added support for user-specified directories to exclude

[Fabio Zanini]

On Fri, Jan 27, 2012 at 12:41:18PM +0200, Tomi Ollila wrote:
> It's easy to combine works together, but more difficult is to choose best 
> terminology: database.exclude vs. new.ignore (or something in between or 
> totally different). Ideas anyone?

I would opt for including two mechanisms for ignoring dirs:
- explicit mention in the config file (e.g. under new.ignore);
- check for presence of a file in the dir (e.g. .notmuchignore), much
  like mu [1].

The second mechanism is useful in case of a large number of folders to
be ignored, as one can simply copy .notmuchignore in all those folders
with a for cycle; listing all of them in the config file would be
cumbersome - and make the config file itself horribly bloated.

Cheers,
Fabio

[1] http://code.google.com/p/mu0/

[PATCH] moved _config_(get|set)_list () functions earlyer in the file

[Tomi Ollila]

Moved static functions _config_get_list () and _config_set_list ()
closer to the beginning of file so that their definition is known
(without adding forward declarations) in upcoming changes.
---

This addresses Ethan's comments. Thanks.

s/_notmuch_/_config_/ and changed 'in further work' to 'in upcoming
changes' -- 'changes' being more generic that 'patches'.
 notmuch-config.c |   84 +++++++++++++++++++++++++++---------------------------
 1 files changed, 42 insertions(+), 42 deletions(-)

diff --git a/notmuch-config.c b/notmuch-config.c
index 0ded6d7..a124e34 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -467,6 +467,48 @@ notmuch_config_save (notmuch_config_t *config)
     return 0;
 }
 
+static const char **
+_config_get_list (notmuch_config_t *config,
+		  const char *section, const char *key,
+		  const char ***outlist, size_t *list_length, size_t *ret_length)
+{
+    assert(outlist);
+
+    if (*outlist == NULL) {
+
+	char **inlist = g_key_file_get_string_list (config->key_file,
+					     section, key, list_length, NULL);
+	if (inlist) {
+	    unsigned int i;
+
+	    *outlist = talloc_size (config, sizeof (char *) * (*list_length + 1));
+
+	    for (i = 0; i < *list_length; i++)
+		(*outlist)[i] = talloc_strdup (*outlist, inlist[i]);
+
+	    (*outlist)[i] = NULL;
+
+	    g_strfreev (inlist);
+	}
+    }
+
+    if (ret_length)
+	*ret_length = *list_length;
+
+    return *outlist;
+}
+
+static void
+_config_set_list (notmuch_config_t *config,
+		  const char *group, const char *name,
+		  const char *list[],
+		  size_t length, const char ***config_var )
+{
+    g_key_file_set_string_list (config->key_file, group, name, list, length);
+    talloc_free (*config_var);
+    *config_var = NULL;
+}
+
 const char *
 notmuch_config_get_database_path (notmuch_config_t *config)
 {
@@ -551,37 +593,6 @@ notmuch_config_set_user_primary_email (notmuch_config_t *config,
     config->user_primary_email = NULL;
 }
 
-static const char **
-_config_get_list (notmuch_config_t *config,
-		  const char *section, const char *key,
-		  const char ***outlist, size_t *list_length, size_t *ret_length)
-{
-    assert(outlist);
-
-    if (*outlist == NULL) {
-
-	char **inlist = g_key_file_get_string_list (config->key_file,
-					     section, key, list_length, NULL);
-	if (inlist) {
-	    unsigned int i;
-
-	    *outlist = talloc_size (config, sizeof (char *) * (*list_length + 1));
-
-	    for (i = 0; i < *list_length; i++)
-		(*outlist)[i] = talloc_strdup (*outlist, inlist[i]);
-
-	    (*outlist)[i] = NULL;
-
-	    g_strfreev (inlist);
-	}
-    }
-
-    if (ret_length)
-	*ret_length = *list_length;
-
-    return *outlist;
-}
-
 const char **
 notmuch_config_get_user_other_email (notmuch_config_t *config,   size_t *length)
 {
@@ -598,17 +609,6 @@ notmuch_config_get_new_tags (notmuch_config_t *config,   size_t *length)
 			     &(config->new_tags_length), length);
 }
 
-static void
-_config_set_list (notmuch_config_t *config,
-		  const char *group, const char *name,
-		  const char *list[],
-		  size_t length, const char ***config_var )
-{
-    g_key_file_set_string_list (config->key_file, group, name, list, length);
-    talloc_free (*config_var);
-    *config_var = NULL;
-}
-
 void
 notmuch_config_set_user_other_email (notmuch_config_t *config,
 				     const char *list[],
-- 
1.7.8.2

Re: [PATCH] moved _config_(get|set)_list () functions earlyer in the file

[Austin Clements]

s/earlyer/earlier/, but unless someone feels strongly about that, I
don't think we need another version of this trivial patch.

Quoth Tomi Ollila on Jan 30 at 12:31 pm:
> Moved static functions _config_get_list () and _config_set_list ()
> closer to the beginning of file so that their definition is known
> (without adding forward declarations) in upcoming changes.
> ---
> 
> This addresses Ethan's comments. Thanks.
> 
> s/_notmuch_/_config_/ and changed 'in further work' to 'in upcoming
> changes' -- 'changes' being more generic that 'patches'.
>  notmuch-config.c |   84 +++++++++++++++++++++++++++---------------------------
>  1 files changed, 42 insertions(+), 42 deletions(-)
> 
> diff --git a/notmuch-config.c b/notmuch-config.c
> index 0ded6d7..a124e34 100644
> --- a/notmuch-config.c
> +++ b/notmuch-config.c
> @@ -467,6 +467,48 @@ notmuch_config_save (notmuch_config_t *config)
>      return 0;
>  }
>  
> +static const char **
> +_config_get_list (notmuch_config_t *config,
> +		  const char *section, const char *key,
> +		  const char ***outlist, size_t *list_length, size_t *ret_length)
> +{
> +    assert(outlist);
> +
> +    if (*outlist == NULL) {
> +
> +	char **inlist = g_key_file_get_string_list (config->key_file,
> +					     section, key, list_length, NULL);
> +	if (inlist) {
> +	    unsigned int i;
> +
> +	    *outlist = talloc_size (config, sizeof (char *) * (*list_length + 1));
> +
> +	    for (i = 0; i < *list_length; i++)
> +		(*outlist)[i] = talloc_strdup (*outlist, inlist[i]);
> +
> +	    (*outlist)[i] = NULL;
> +
> +	    g_strfreev (inlist);
> +	}
> +    }
> +
> +    if (ret_length)
> +	*ret_length = *list_length;
> +
> +    return *outlist;
> +}
> +
> +static void
> +_config_set_list (notmuch_config_t *config,
> +		  const char *group, const char *name,
> +		  const char *list[],
> +		  size_t length, const char ***config_var )
> +{
> +    g_key_file_set_string_list (config->key_file, group, name, list, length);
> +    talloc_free (*config_var);
> +    *config_var = NULL;
> +}
> +
>  const char *
>  notmuch_config_get_database_path (notmuch_config_t *config)
>  {
> @@ -551,37 +593,6 @@ notmuch_config_set_user_primary_email (notmuch_config_t *config,
>      config->user_primary_email = NULL;
>  }
>  
> -static const char **
> -_config_get_list (notmuch_config_t *config,
> -		  const char *section, const char *key,
> -		  const char ***outlist, size_t *list_length, size_t *ret_length)
> -{
> -    assert(outlist);
> -
> -    if (*outlist == NULL) {
> -
> -	char **inlist = g_key_file_get_string_list (config->key_file,
> -					     section, key, list_length, NULL);
> -	if (inlist) {
> -	    unsigned int i;
> -
> -	    *outlist = talloc_size (config, sizeof (char *) * (*list_length + 1));
> -
> -	    for (i = 0; i < *list_length; i++)
> -		(*outlist)[i] = talloc_strdup (*outlist, inlist[i]);
> -
> -	    (*outlist)[i] = NULL;
> -
> -	    g_strfreev (inlist);
> -	}
> -    }
> -
> -    if (ret_length)
> -	*ret_length = *list_length;
> -
> -    return *outlist;
> -}
> -
>  const char **
>  notmuch_config_get_user_other_email (notmuch_config_t *config,   size_t *length)
>  {
> @@ -598,17 +609,6 @@ notmuch_config_get_new_tags (notmuch_config_t *config,   size_t *length)
>  			     &(config->new_tags_length), length);
>  }
>  
> -static void
> -_config_set_list (notmuch_config_t *config,
> -		  const char *group, const char *name,
> -		  const char *list[],
> -		  size_t length, const char ***config_var )
> -{
> -    g_key_file_set_string_list (config->key_file, group, name, list, length);
> -    talloc_free (*config_var);
> -    *config_var = NULL;
> -}
> -
>  void
>  notmuch_config_set_user_other_email (notmuch_config_t *config,
>  				     const char *list[],

Re: [PATCH] moved _config_(get|set)_list () functions earlyer in the file

[David Bremner]

On Mon, 30 Jan 2012 12:31:25 +0200, Tomi Ollila <tomi.ollila@iki.fi> wrote:
> Moved static functions _config_get_list () and _config_set_list ()
> closer to the beginning of file so that their definition is known
> (without adding forward declarations) in upcoming changes.
> ---

pushed,

d

[PATCH] added support for user-specified files & directories to ignore

[Tomi Ollila]

A new configuration key 'new.ignore' is used to determine which
files and directories user wants not to be scanned as new mails.

This work merges my previous attempts and Andreas Amann's work
in id:"ylp7hi23mw8.fsf@tyndall.ie"

See notes in id:"20120131-new-ignore-1-git-send-email-too@iki.fi"
---
Notes

1) Currently there is comment for new.ignore in newly created configuration
file but as the list is initially empty there will be not tag in place.

2) Whenever some already existing directory is added to the exclude list
and the parent directory timestamp has not changed, notmuch new will not
notice the directory has gone (as it still is there), user needs to 'touch'
the parent directory before next 'notmuch new' no make notmuch notice.

2012-01-26: could notmuch track mtime of the configuration file and if
that changes, ignore mail directory timestamps ?


3) in id:"1327572718-13411-2-git-send-email-tomi.ollila@iki.fi" dropped...

 notmuch-client.h |    8 ++++++++
 notmuch-config.c |   35 +++++++++++++++++++++++++++++++++--
 notmuch-new.c    |   45 +++++++++++++++++++++++++++++++++------------
 3 files changed, 74 insertions(+), 14 deletions(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index e0eb594..c62ce78 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -250,6 +250,14 @@ notmuch_config_set_new_tags (notmuch_config_t *config,
 			     const char *new_tags[],
 			     size_t length);
 
+const char **
+notmuch_config_get_new_ignore (notmuch_config_t *config,
+			       size_t *length);
+void
+notmuch_config_set_new_ignore (notmuch_config_t *config,
+			       const char *new_ignore[],
+			       size_t length);
+
 notmuch_bool_t
 notmuch_config_get_maildir_synchronize_flags (notmuch_config_t *config);
 
diff --git a/notmuch-config.c b/notmuch-config.c
index a124e34..f1cc5c2 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -44,7 +44,10 @@ static const char new_config_comment[] =
     " The following options are supported here:\n"
     "\n"
     "\ttags	A list (separated by ';') of the tags that will be\n"
-    "\t	added to all messages incorporated by \"notmuch new\".\n";
+    "\t	added to all messages incorporated by \"notmuch new\".\n"
+    "\n"
+    "\tignore	A list (separated by ';') of files and directories that"
+    "\t	will not be searched for messages by \"notmuch new\".\n";
 
 static const char user_config_comment[] =
     " User configuration\n"
@@ -105,6 +108,8 @@ struct _notmuch_config {
     size_t user_other_email_length;
     const char **new_tags;
     size_t new_tags_length;
+    const char **new_ignore;
+    size_t new_ignore_length;
     notmuch_bool_t maildir_synchronize_flags;
     const char **search_exclude_tags;
     size_t search_exclude_tags_length;
@@ -264,6 +269,8 @@ notmuch_config_open (void *ctx,
     config->user_other_email_length = 0;
     config->new_tags = NULL;
     config->new_tags_length = 0;
+    config->new_ignore = NULL;
+    config->new_ignore_length = 0;
     config->maildir_synchronize_flags = TRUE;
     config->search_exclude_tags = NULL;
     config->search_exclude_tags_length = 0;
@@ -360,7 +367,11 @@ notmuch_config_open (void *ctx,
         const char *tags[] = { "unread", "inbox" };
 	notmuch_config_set_new_tags (config, tags, 2);
     }
-
+#if 0 /* No point setting empty list -- it's not written */
+    if (notmuch_config_get_new_ignore (config, &tmp) == NULL) {
+	notmuch_config_set_new_ignore (config, NULL, 0);
+    }
+#endif
     if (notmuch_config_get_search_exclude_tags (config, &tmp) == NULL) {
 	if (is_new) {
 	    const char *tags[] = { "deleted", "spam" };
@@ -609,6 +620,15 @@ notmuch_config_get_new_tags (notmuch_config_t *config,   size_t *length)
 			     &(config->new_tags_length), length);
 }
 
+const char **
+notmuch_config_get_new_ignore (notmuch_config_t *config,   size_t *length)
+{
+    return _config_get_list (config, "new", "ignore",
+			     &(config->new_ignore),
+			     &(config->new_ignore_length), length);
+}
+
+
 void
 notmuch_config_set_user_other_email (notmuch_config_t *config,
 				     const char *list[],
@@ -627,6 +647,17 @@ notmuch_config_set_new_tags (notmuch_config_t *config,
 		     &(config->new_tags));
 }
 
+#if 0 /* UNNEEDED SO FAR */
+void
+notmuch_config_set_new_ignore (notmuch_config_t *config,
+			       const char *list[],
+			       size_t length)
+{
+    _config_set_list (config, "new", "ignore", list, length,
+		     &(config->new_ignore));
+}
+#endif
+
 const char **
 notmuch_config_get_search_exclude_tags (notmuch_config_t *config, size_t *length)
 {
diff --git a/notmuch-new.c b/notmuch-new.c
index a569a54..36d5c5d 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -39,6 +39,8 @@ typedef struct {
     int verbose;
     const char **new_tags;
     size_t new_tags_length;
+    const char **new_ignore;
+    size_t new_ignore_length;
 
     int total_files;
     int processed_files;
@@ -181,6 +183,20 @@ _entries_resemble_maildir (struct dirent **entries, int count)
     return 0;
 }
 
+/* Check if user asked to ignore these files/directories */
+
+static int
+_entry_in_ignore_list (const char *entry, add_files_state_t *state)
+{
+    size_t i, ignore_length = state->new_ignore_length;
+
+    for (i = 0; i < ignore_length; i++)
+	if (strcmp (entry, state->new_ignore[i]) == 0)
+	    return 1;
+
+    return 0;
+}
+
 /* Examine 'path' recursively as follows:
  *
  *   o Ask the filesystem for the mtime of 'path' (fs_mtime)
@@ -320,15 +336,15 @@ add_files_recursive (notmuch_database_t *notmuch,
 	}
 
 	/* Ignore special directories to avoid infinite recursion.
-	 * Also ignore the .notmuch directory and any "tmp" directory
-	 * that appears within a maildir.
+	 * Also ignore the .notmuch directory, any "tmp" directory
+	 * that appears within a maildir and files/directories
+	 * user have configured to be ignored.
 	 */
-	/* XXX: Eventually we'll want more sophistication to let the
-	 * user specify files to be ignored. */
 	if (strcmp (entry->d_name, ".") == 0 ||
 	    strcmp (entry->d_name, "..") == 0 ||
 	    (is_maildir && strcmp (entry->d_name, "tmp") == 0) ||
-	    strcmp (entry->d_name, ".notmuch") ==0)
+	    strcmp (entry->d_name, ".notmuch") == 0 ||
+	    _entry_in_ignore_list (entry->d_name, state))
 	{
 	    continue;
 	}
@@ -369,6 +385,10 @@ add_files_recursive (notmuch_database_t *notmuch,
 
         entry = fs_entries[i];
 
+	/* Ignore files & directories user has configured to be ignored */
+	if (_entry_in_ignore_list (entry->d_name, state))
+	    continue;
+
 	/* Check if we've walked past any names in db_files or
 	 * db_subdirs. If so, these have been deleted. */
 	while (notmuch_filenames_valid (db_files) &&
@@ -648,7 +668,7 @@ add_files (notmuch_database_t *notmuch,
  * initialized to zero by the top-level caller before calling
  * count_files). */
 static void
-count_files (const char *path, int *count)
+count_files (const char *path, int *count, add_files_state_t *state)
 {
     struct dirent *entry = NULL;
     char *next;
@@ -670,13 +690,13 @@ count_files (const char *path, int *count)
         entry = fs_entries[i++];
 
 	/* Ignore special directories to avoid infinite recursion.
-	 * Also ignore the .notmuch directory.
+	 * Also ignore the .notmuch directory and files/directories
+	 * user have configured to be ignored.
 	 */
-	/* XXX: Eventually we'll want more sophistication to let the
-	 * user specify files to be ignored. */
 	if (strcmp (entry->d_name, ".") == 0 ||
 	    strcmp (entry->d_name, "..") == 0 ||
-	    strcmp (entry->d_name, ".notmuch") == 0)
+	    strcmp (entry->d_name, ".notmuch") == 0 ||
+	    _entry_in_ignore_list (entry->d_name, state))
 	{
 	    continue;
 	}
@@ -697,7 +717,7 @@ count_files (const char *path, int *count)
 		fflush (stdout);
 	    }
 	} else if (S_ISDIR (st.st_mode)) {
-	    count_files (next, count);
+	    count_files (next, count, state);
 	}
 
 	free (next);
@@ -837,6 +857,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
 	return 1;
 
     add_files_state.new_tags = notmuch_config_get_new_tags (config, &add_files_state.new_tags_length);
+    add_files_state.new_ignore = notmuch_config_get_new_ignore (config, &add_files_state.new_ignore_length);
     add_files_state.synchronize_flags = notmuch_config_get_maildir_synchronize_flags (config);
     db_path = notmuch_config_get_database_path (config);
 
@@ -852,7 +873,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
 	int count;
 
 	count = 0;
-	count_files (db_path, &count);
+	count_files (db_path, &count, &add_files_state);
 	if (interrupted)
 	    return 1;
 
-- 
1.7.6.5

[PATCH] test: add tests wrt ignoring user-specified files and directories

[Pieter Praet]

Files and directories which are specified in 'new.ignore' in the
config file shouldn't be indexed nor reported by `notmuch new'.

---
 test/new |   23 +++++++++++++++++++++++
 1 files changed, 23 insertions(+), 0 deletions(-)

diff --git a/test/new b/test/new
index 49f390d..740ba05 100755
--- a/test/new
+++ b/test/new
@@ -153,4 +153,27 @@ rm -rf "${MAIL_DIR}"/two
 output=$(NOTMUCH_NEW)
 test_expect_equal "$output" "No new mail. Removed 3 messages."
 
+test_begin_subtest "Skip and report non-mail files"
+generate_message
+mkdir -p "${MAIL_DIR}"/.git && touch "${MAIL_DIR}"/.git/config
+touch "${MAIL_DIR}"/ignored_file
+touch "${MAIL_DIR}"/.ignored_hidden_file
+output=$(NOTMUCH_NEW 2>&1)
+test_expect_equal "$output" \
+"Note: Ignoring non-mail file: ${MAIL_DIR}/.git/config
+Note: Ignoring non-mail file: ${MAIL_DIR}/.ignored_hidden_file
+Note: Ignoring non-mail file: ${MAIL_DIR}/ignored_file
+Added 1 new message to the database."
+
+test_begin_subtest "Ignore files and directories specified in new.ignore"
+test_subtest_known_broken
+generate_message
+notmuch config set new.ignore .git ignored_file .ignored_hidden_file
+mkdir -p "${MAIL_DIR}"/.git && touch "${MAIL_DIR}"/.git/config
+touch "${MAIL_DIR}"/ignored_file
+touch "${MAIL_DIR}"/.ignored_hidden_file
+output=$(NOTMUCH_NEW 2>&1)
+test_expect_equal "$output" "Added 1 new message to the database."
+
+
 test_done
-- 
1.7.8.1

Re: [PATCH] added support for user-specified files & directories to ignore

[Pieter Praet]

On Tue, 31 Jan 2012 18:28:04 +0200, Tomi Ollila <tomi.ollila@iki.fi> wrote:
> A new configuration key 'new.ignore' is used to determine which
> files and directories user wants not to be scanned as new mails.
> 
> This work merges my previous attempts and Andreas Amann's work
> in id:"ylp7hi23mw8.fsf@tyndall.ie"
> 
> See notes in id:"20120131-new-ignore-1-git-send-email-too@iki.fi"
> ---

Great work Tomi!

LGTM (though keep in mind, I'm no C coder).

> Notes
> 
> 1) Currently there is comment for new.ignore in newly created configuration
> file but as the list is initially empty there will be not tag in place.
> 
> 2) Whenever some already existing directory is added to the exclude list
> and the parent directory timestamp has not changed, notmuch new will not
> notice the directory has gone (as it still is there), user needs to 'touch'
> the parent directory before next 'notmuch new' no make notmuch notice.
> 
> 2012-01-26: could notmuch track mtime of the configuration file and if
> that changes, ignore mail directory timestamps ?
> 
> 
> 3) in id:"1327572718-13411-2-git-send-email-tomi.ollila@iki.fi" dropped...
> 
>  notmuch-client.h |    8 ++++++++
>  notmuch-config.c |   35 +++++++++++++++++++++++++++++++++--
>  notmuch-new.c    |   45 +++++++++++++++++++++++++++++++++------------
>  3 files changed, 74 insertions(+), 14 deletions(-)
> 
> diff --git a/notmuch-client.h b/notmuch-client.h
> index e0eb594..c62ce78 100644
> --- a/notmuch-client.h
> +++ b/notmuch-client.h
> @@ -250,6 +250,14 @@ notmuch_config_set_new_tags (notmuch_config_t *config,
>  			     const char *new_tags[],
>  			     size_t length);
>  
> +const char **
> +notmuch_config_get_new_ignore (notmuch_config_t *config,
> +			       size_t *length);
> +void
> +notmuch_config_set_new_ignore (notmuch_config_t *config,
> +			       const char *new_ignore[],
> +			       size_t length);
> +
>  notmuch_bool_t
>  notmuch_config_get_maildir_synchronize_flags (notmuch_config_t *config);
>  
> diff --git a/notmuch-config.c b/notmuch-config.c
> index a124e34..f1cc5c2 100644
> --- a/notmuch-config.c
> +++ b/notmuch-config.c
> @@ -44,7 +44,10 @@ static const char new_config_comment[] =
>      " The following options are supported here:\n"
>      "\n"
>      "\ttags	A list (separated by ';') of the tags that will be\n"
> -    "\t	added to all messages incorporated by \"notmuch new\".\n";
> +    "\t	added to all messages incorporated by \"notmuch new\".\n"
> +    "\n"
> +    "\tignore	A list (separated by ';') of files and directories that"
> +    "\t	will not be searched for messages by \"notmuch new\".\n";
>  
>  static const char user_config_comment[] =
>      " User configuration\n"
> @@ -105,6 +108,8 @@ struct _notmuch_config {
>      size_t user_other_email_length;
>      const char **new_tags;
>      size_t new_tags_length;
> +    const char **new_ignore;
> +    size_t new_ignore_length;
>      notmuch_bool_t maildir_synchronize_flags;
>      const char **search_exclude_tags;
>      size_t search_exclude_tags_length;
> @@ -264,6 +269,8 @@ notmuch_config_open (void *ctx,
>      config->user_other_email_length = 0;
>      config->new_tags = NULL;
>      config->new_tags_length = 0;
> +    config->new_ignore = NULL;
> +    config->new_ignore_length = 0;
>      config->maildir_synchronize_flags = TRUE;
>      config->search_exclude_tags = NULL;
>      config->search_exclude_tags_length = 0;
> @@ -360,7 +367,11 @@ notmuch_config_open (void *ctx,
>          const char *tags[] = { "unread", "inbox" };
>  	notmuch_config_set_new_tags (config, tags, 2);
>      }
> -
> +#if 0 /* No point setting empty list -- it's not written */
> +    if (notmuch_config_get_new_ignore (config, &tmp) == NULL) {
> +	notmuch_config_set_new_ignore (config, NULL, 0);
> +    }
> +#endif
>      if (notmuch_config_get_search_exclude_tags (config, &tmp) == NULL) {
>  	if (is_new) {
>  	    const char *tags[] = { "deleted", "spam" };
> @@ -609,6 +620,15 @@ notmuch_config_get_new_tags (notmuch_config_t *config,   size_t *length)
>  			     &(config->new_tags_length), length);
>  }
>  
> +const char **
> +notmuch_config_get_new_ignore (notmuch_config_t *config,   size_t *length)
> +{
> +    return _config_get_list (config, "new", "ignore",
> +			     &(config->new_ignore),
> +			     &(config->new_ignore_length), length);
> +}
> +
> +
>  void
>  notmuch_config_set_user_other_email (notmuch_config_t *config,
>  				     const char *list[],
> @@ -627,6 +647,17 @@ notmuch_config_set_new_tags (notmuch_config_t *config,
>  		     &(config->new_tags));
>  }
>  
> +#if 0 /* UNNEEDED SO FAR */
> +void
> +notmuch_config_set_new_ignore (notmuch_config_t *config,
> +			       const char *list[],
> +			       size_t length)
> +{
> +    _config_set_list (config, "new", "ignore", list, length,
> +		     &(config->new_ignore));
> +}
> +#endif
> +

Is this really discarded during compilation ?!?

The results of my test [1] suggest otherwise...


>  const char **
>  notmuch_config_get_search_exclude_tags (notmuch_config_t *config, size_t *length)
>  {
> diff --git a/notmuch-new.c b/notmuch-new.c
> index a569a54..36d5c5d 100644
> --- a/notmuch-new.c
> +++ b/notmuch-new.c
> @@ -39,6 +39,8 @@ typedef struct {
>      int verbose;
>      const char **new_tags;
>      size_t new_tags_length;
> +    const char **new_ignore;
> +    size_t new_ignore_length;
>  
>      int total_files;
>      int processed_files;
> @@ -181,6 +183,20 @@ _entries_resemble_maildir (struct dirent **entries, int count)
>      return 0;
>  }
>  
> +/* Check if user asked to ignore these files/directories */
> +
> +static int
> +_entry_in_ignore_list (const char *entry, add_files_state_t *state)
> +{
> +    size_t i, ignore_length = state->new_ignore_length;
> +
> +    for (i = 0; i < ignore_length; i++)
> +	if (strcmp (entry, state->new_ignore[i]) == 0)
> +	    return 1;
> +
> +    return 0;
> +}
> +
>  /* Examine 'path' recursively as follows:
>   *
>   *   o Ask the filesystem for the mtime of 'path' (fs_mtime)
> @@ -320,15 +336,15 @@ add_files_recursive (notmuch_database_t *notmuch,
>  	}
>  
>  	/* Ignore special directories to avoid infinite recursion.
> -	 * Also ignore the .notmuch directory and any "tmp" directory
> -	 * that appears within a maildir.
> +	 * Also ignore the .notmuch directory, any "tmp" directory
> +	 * that appears within a maildir and files/directories
> +	 * user have configured to be ignored.
>  	 */
> -	/* XXX: Eventually we'll want more sophistication to let the
> -	 * user specify files to be ignored. */
>  	if (strcmp (entry->d_name, ".") == 0 ||
>  	    strcmp (entry->d_name, "..") == 0 ||
>  	    (is_maildir && strcmp (entry->d_name, "tmp") == 0) ||
> -	    strcmp (entry->d_name, ".notmuch") ==0)
> +	    strcmp (entry->d_name, ".notmuch") == 0 ||
> +	    _entry_in_ignore_list (entry->d_name, state))
>  	{
>  	    continue;
>  	}
> @@ -369,6 +385,10 @@ add_files_recursive (notmuch_database_t *notmuch,
>  
>          entry = fs_entries[i];
>  
> +	/* Ignore files & directories user has configured to be ignored */
> +	if (_entry_in_ignore_list (entry->d_name, state))
> +	    continue;
> +
>  	/* Check if we've walked past any names in db_files or
>  	 * db_subdirs. If so, these have been deleted. */
>  	while (notmuch_filenames_valid (db_files) &&
> @@ -648,7 +668,7 @@ add_files (notmuch_database_t *notmuch,
>   * initialized to zero by the top-level caller before calling
>   * count_files). */
>  static void
> -count_files (const char *path, int *count)
> +count_files (const char *path, int *count, add_files_state_t *state)
>  {
>      struct dirent *entry = NULL;
>      char *next;
> @@ -670,13 +690,13 @@ count_files (const char *path, int *count)
>          entry = fs_entries[i++];
>  
>  	/* Ignore special directories to avoid infinite recursion.
> -	 * Also ignore the .notmuch directory.
> +	 * Also ignore the .notmuch directory and files/directories
> +	 * user have configured to be ignored.
>  	 */
> -	/* XXX: Eventually we'll want more sophistication to let the
> -	 * user specify files to be ignored. */
>  	if (strcmp (entry->d_name, ".") == 0 ||
>  	    strcmp (entry->d_name, "..") == 0 ||
> -	    strcmp (entry->d_name, ".notmuch") == 0)
> +	    strcmp (entry->d_name, ".notmuch") == 0 ||
> +	    _entry_in_ignore_list (entry->d_name, state))
>  	{
>  	    continue;
>  	}
> @@ -697,7 +717,7 @@ count_files (const char *path, int *count)
>  		fflush (stdout);
>  	    }
>  	} else if (S_ISDIR (st.st_mode)) {
> -	    count_files (next, count);
> +	    count_files (next, count, state);
>  	}
>  
>  	free (next);
> @@ -837,6 +857,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
>  	return 1;
>  
>      add_files_state.new_tags = notmuch_config_get_new_tags (config, &add_files_state.new_tags_length);
> +    add_files_state.new_ignore = notmuch_config_get_new_ignore (config, &add_files_state.new_ignore_length);
>      add_files_state.synchronize_flags = notmuch_config_get_maildir_synchronize_flags (config);
>      db_path = notmuch_config_get_database_path (config);
>  
> @@ -852,7 +873,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
>  	int count;
>  
>  	count = 0;
> -	count_files (db_path, &count);
> +	count_files (db_path, &count, &add_files_state);
>  	if (interrupted)
>  	    return 1;
>  
> -- 
> 1.7.6.5
> 
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch


Peace

-- 
Pieter

[1] id:"1328105573-4626-1-git-send-email-pieter@praet.org"

Re: [PATCH] added support for user-specified files & directories to ignore

[Jani Nikula]

On Tue, 31 Jan 2012 18:28:04 +0200, Tomi Ollila <tomi.ollila@iki.fi> wrote:
> A new configuration key 'new.ignore' is used to determine which
> files and directories user wants not to be scanned as new mails.

Hi, please find a few drive-by "while my code is compiling" nitpick
review comments inline. ;) Mostly looks good.

BR,
Jani.


> 
> This work merges my previous attempts and Andreas Amann's work
> in id:"ylp7hi23mw8.fsf@tyndall.ie"
> 
> See notes in id:"20120131-new-ignore-1-git-send-email-too@iki.fi"
> ---
> Notes
> 
> 1) Currently there is comment for new.ignore in newly created configuration
> file but as the list is initially empty there will be not tag in place.
> 
> 2) Whenever some already existing directory is added to the exclude list
> and the parent directory timestamp has not changed, notmuch new will not
> notice the directory has gone (as it still is there), user needs to 'touch'
> the parent directory before next 'notmuch new' no make notmuch notice.
> 
> 2012-01-26: could notmuch track mtime of the configuration file and if
> that changes, ignore mail directory timestamps ?
> 
> 
> 3) in id:"1327572718-13411-2-git-send-email-tomi.ollila@iki.fi" dropped...
> 
>  notmuch-client.h |    8 ++++++++
>  notmuch-config.c |   35 +++++++++++++++++++++++++++++++++--
>  notmuch-new.c    |   45 +++++++++++++++++++++++++++++++++------------
>  3 files changed, 74 insertions(+), 14 deletions(-)
> 
> diff --git a/notmuch-client.h b/notmuch-client.h
> index e0eb594..c62ce78 100644
> --- a/notmuch-client.h
> +++ b/notmuch-client.h
> @@ -250,6 +250,14 @@ notmuch_config_set_new_tags (notmuch_config_t *config,
>  			     const char *new_tags[],
>  			     size_t length);
>  
> +const char **
> +notmuch_config_get_new_ignore (notmuch_config_t *config,
> +			       size_t *length);
> +void
> +notmuch_config_set_new_ignore (notmuch_config_t *config,
> +			       const char *new_ignore[],
> +			       size_t length);
> +
>  notmuch_bool_t
>  notmuch_config_get_maildir_synchronize_flags (notmuch_config_t *config);
>  
> diff --git a/notmuch-config.c b/notmuch-config.c
> index a124e34..f1cc5c2 100644
> --- a/notmuch-config.c
> +++ b/notmuch-config.c
> @@ -44,7 +44,10 @@ static const char new_config_comment[] =
>      " The following options are supported here:\n"
>      "\n"
>      "\ttags	A list (separated by ';') of the tags that will be\n"
> -    "\t	added to all messages incorporated by \"notmuch new\".\n";
> +    "\t	added to all messages incorporated by \"notmuch new\".\n"
> +    "\n"
> +    "\tignore	A list (separated by ';') of files and directories that"
> +    "\t	will not be searched for messages by \"notmuch new\".\n";
>  
>  static const char user_config_comment[] =
>      " User configuration\n"
> @@ -105,6 +108,8 @@ struct _notmuch_config {
>      size_t user_other_email_length;
>      const char **new_tags;
>      size_t new_tags_length;
> +    const char **new_ignore;
> +    size_t new_ignore_length;
>      notmuch_bool_t maildir_synchronize_flags;
>      const char **search_exclude_tags;
>      size_t search_exclude_tags_length;
> @@ -264,6 +269,8 @@ notmuch_config_open (void *ctx,
>      config->user_other_email_length = 0;
>      config->new_tags = NULL;
>      config->new_tags_length = 0;
> +    config->new_ignore = NULL;
> +    config->new_ignore_length = 0;
>      config->maildir_synchronize_flags = TRUE;
>      config->search_exclude_tags = NULL;
>      config->search_exclude_tags_length = 0;
> @@ -360,7 +367,11 @@ notmuch_config_open (void *ctx,
>          const char *tags[] = { "unread", "inbox" };
>  	notmuch_config_set_new_tags (config, tags, 2);
>      }
> -
> +#if 0 /* No point setting empty list -- it's not written */
> +    if (notmuch_config_get_new_ignore (config, &tmp) == NULL) {
> +	notmuch_config_set_new_ignore (config, NULL, 0);
> +    }
> +#endif

I'd prefer not having any #if 0 code. How about just having the code
there, even if it ends up being a NOP? You never know when the config
plumbing is going to change (it could use some love), e.g. it might
write the default value in comments in the future.

>      if (notmuch_config_get_search_exclude_tags (config, &tmp) == NULL) {
>  	if (is_new) {
>  	    const char *tags[] = { "deleted", "spam" };
> @@ -609,6 +620,15 @@ notmuch_config_get_new_tags (notmuch_config_t *config,   size_t *length)
>  			     &(config->new_tags_length), length);
>  }
>  
> +const char **
> +notmuch_config_get_new_ignore (notmuch_config_t *config,   size_t *length)

Is there extra space in there?

> +{
> +    return _config_get_list (config, "new", "ignore",
> +			     &(config->new_ignore),
> +			     &(config->new_ignore_length), length);
> +}
> +
> +
>  void
>  notmuch_config_set_user_other_email (notmuch_config_t *config,
>  				     const char *list[],
> @@ -627,6 +647,17 @@ notmuch_config_set_new_tags (notmuch_config_t *config,
>  		     &(config->new_tags));
>  }
>  
> +#if 0 /* UNNEEDED SO FAR */
> +void
> +notmuch_config_set_new_ignore (notmuch_config_t *config,
> +			       const char *list[],
> +			       size_t length)
> +{
> +    _config_set_list (config, "new", "ignore", list, length,
> +		     &(config->new_ignore));
> +}
> +#endif

Same here, just remove the #if 0.

> +
>  const char **
>  notmuch_config_get_search_exclude_tags (notmuch_config_t *config, size_t *length)
>  {
> diff --git a/notmuch-new.c b/notmuch-new.c
> index a569a54..36d5c5d 100644
> --- a/notmuch-new.c
> +++ b/notmuch-new.c
> @@ -39,6 +39,8 @@ typedef struct {
>      int verbose;
>      const char **new_tags;
>      size_t new_tags_length;
> +    const char **new_ignore;
> +    size_t new_ignore_length;
>  
>      int total_files;
>      int processed_files;
> @@ -181,6 +183,20 @@ _entries_resemble_maildir (struct dirent **entries, int count)
>      return 0;
>  }
>  
> +/* Check if user asked to ignore these files/directories */
> +
> +static int

notmuch_bool_t?

> +_entry_in_ignore_list (const char *entry, add_files_state_t *state)
> +{
> +    size_t i, ignore_length = state->new_ignore_length;

ignore_length is an unnecessary temp variable.

> +
> +    for (i = 0; i < ignore_length; i++)
> +	if (strcmp (entry, state->new_ignore[i]) == 0)
> +	    return 1;
> +
> +    return 0;
> +}
> +
>  /* Examine 'path' recursively as follows:
>   *
>   *   o Ask the filesystem for the mtime of 'path' (fs_mtime)
> @@ -320,15 +336,15 @@ add_files_recursive (notmuch_database_t *notmuch,
>  	}
>  
>  	/* Ignore special directories to avoid infinite recursion.
> -	 * Also ignore the .notmuch directory and any "tmp" directory
> -	 * that appears within a maildir.
> +	 * Also ignore the .notmuch directory, any "tmp" directory
> +	 * that appears within a maildir and files/directories
> +	 * user have configured to be ignored.

s/user have/the user has/ ?

>  	 */
> -	/* XXX: Eventually we'll want more sophistication to let the
> -	 * user specify files to be ignored. */
>  	if (strcmp (entry->d_name, ".") == 0 ||
>  	    strcmp (entry->d_name, "..") == 0 ||
>  	    (is_maildir && strcmp (entry->d_name, "tmp") == 0) ||
> -	    strcmp (entry->d_name, ".notmuch") ==0)
> +	    strcmp (entry->d_name, ".notmuch") == 0 ||
> +	    _entry_in_ignore_list (entry->d_name, state))
>  	{
>  	    continue;
>  	}
> @@ -369,6 +385,10 @@ add_files_recursive (notmuch_database_t *notmuch,
>  
>          entry = fs_entries[i];
>  
> +	/* Ignore files & directories user has configured to be ignored */
> +	if (_entry_in_ignore_list (entry->d_name, state))
> +	    continue;
> +
>  	/* Check if we've walked past any names in db_files or
>  	 * db_subdirs. If so, these have been deleted. */
>  	while (notmuch_filenames_valid (db_files) &&
> @@ -648,7 +668,7 @@ add_files (notmuch_database_t *notmuch,
>   * initialized to zero by the top-level caller before calling
>   * count_files). */
>  static void
> -count_files (const char *path, int *count)
> +count_files (const char *path, int *count, add_files_state_t *state)
>  {
>      struct dirent *entry = NULL;
>      char *next;
> @@ -670,13 +690,13 @@ count_files (const char *path, int *count)
>          entry = fs_entries[i++];
>  
>  	/* Ignore special directories to avoid infinite recursion.
> -	 * Also ignore the .notmuch directory.
> +	 * Also ignore the .notmuch directory and files/directories
> +	 * user have configured to be ignored.

Same as above.

>  	 */
> -	/* XXX: Eventually we'll want more sophistication to let the
> -	 * user specify files to be ignored. */

Oh, you think this is enough sophistication! ;)

>  	if (strcmp (entry->d_name, ".") == 0 ||
>  	    strcmp (entry->d_name, "..") == 0 ||
> -	    strcmp (entry->d_name, ".notmuch") == 0)
> +	    strcmp (entry->d_name, ".notmuch") == 0 ||
> +	    _entry_in_ignore_list (entry->d_name, state))
>  	{
>  	    continue;
>  	}
> @@ -697,7 +717,7 @@ count_files (const char *path, int *count)
>  		fflush (stdout);
>  	    }
>  	} else if (S_ISDIR (st.st_mode)) {
> -	    count_files (next, count);
> +	    count_files (next, count, state);
>  	}
>  
>  	free (next);
> @@ -837,6 +857,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
>  	return 1;
>  
>      add_files_state.new_tags = notmuch_config_get_new_tags (config, &add_files_state.new_tags_length);
> +    add_files_state.new_ignore = notmuch_config_get_new_ignore (config, &add_files_state.new_ignore_length);
>      add_files_state.synchronize_flags = notmuch_config_get_maildir_synchronize_flags (config);
>      db_path = notmuch_config_get_database_path (config);
>  
> @@ -852,7 +873,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
>  	int count;
>  
>  	count = 0;
> -	count_files (db_path, &count);
> +	count_files (db_path, &count, &add_files_state);
>  	if (interrupted)
>  	    return 1;
>  
> -- 
> 1.7.6.5
> 
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] added support for user-specified files & directories to ignore

[Austin Clements]

Quoth Tomi Ollila on Jan 31 at  6:28 pm:
> A new configuration key 'new.ignore' is used to determine which
> files and directories user wants not to be scanned as new mails.
> 
> This work merges my previous attempts and Andreas Amann's work
> in id:"ylp7hi23mw8.fsf@tyndall.ie"
> 
> See notes in id:"20120131-new-ignore-1-git-send-email-too@iki.fi"
> ---
> Notes
> 
> 1) Currently there is comment for new.ignore in newly created configuration
> file but as the list is initially empty there will be not tag in place.
> 
> 2) Whenever some already existing directory is added to the exclude list
> and the parent directory timestamp has not changed, notmuch new will not
> notice the directory has gone (as it still is there), user needs to 'touch'
> the parent directory before next 'notmuch new' no make notmuch notice.
> 
> 2012-01-26: could notmuch track mtime of the configuration file and if
> that changes, ignore mail directory timestamps ?
> 
> 
> 3) in id:"1327572718-13411-2-git-send-email-tomi.ollila@iki.fi" dropped...
> 
>  notmuch-client.h |    8 ++++++++
>  notmuch-config.c |   35 +++++++++++++++++++++++++++++++++--
>  notmuch-new.c    |   45 +++++++++++++++++++++++++++++++++------------
>  3 files changed, 74 insertions(+), 14 deletions(-)
> 
> diff --git a/notmuch-client.h b/notmuch-client.h
> index e0eb594..c62ce78 100644
> --- a/notmuch-client.h
> +++ b/notmuch-client.h
> @@ -250,6 +250,14 @@ notmuch_config_set_new_tags (notmuch_config_t *config,
>  			     const char *new_tags[],
>  			     size_t length);
>  
> +const char **
> +notmuch_config_get_new_ignore (notmuch_config_t *config,
> +			       size_t *length);

Missing blank line.

> +void
> +notmuch_config_set_new_ignore (notmuch_config_t *config,
> +			       const char *new_ignore[],
> +			       size_t length);
> +
>  notmuch_bool_t
>  notmuch_config_get_maildir_synchronize_flags (notmuch_config_t *config);
>  
> diff --git a/notmuch-config.c b/notmuch-config.c
> index a124e34..f1cc5c2 100644
> --- a/notmuch-config.c
> +++ b/notmuch-config.c
> @@ -44,7 +44,10 @@ static const char new_config_comment[] =
>      " The following options are supported here:\n"
>      "\n"
>      "\ttags	A list (separated by ';') of the tags that will be\n"
> -    "\t	added to all messages incorporated by \"notmuch new\".\n";
> +    "\t	added to all messages incorporated by \"notmuch new\".\n"
> +    "\n"
> +    "\tignore	A list (separated by ';') of files and directories that"
> +    "\t	will not be searched for messages by \"notmuch new\".\n";

This can be interpreted in a lot of ways (e.g., absolute paths, paths
relative to database.path, etc) but is expecting something very
specific.  Maybe "file and directory names"?

>  
>  static const char user_config_comment[] =
>      " User configuration\n"
> @@ -105,6 +108,8 @@ struct _notmuch_config {
>      size_t user_other_email_length;
>      const char **new_tags;
>      size_t new_tags_length;
> +    const char **new_ignore;
> +    size_t new_ignore_length;
>      notmuch_bool_t maildir_synchronize_flags;
>      const char **search_exclude_tags;
>      size_t search_exclude_tags_length;
> @@ -264,6 +269,8 @@ notmuch_config_open (void *ctx,
>      config->user_other_email_length = 0;
>      config->new_tags = NULL;
>      config->new_tags_length = 0;
> +    config->new_ignore = NULL;
> +    config->new_ignore_length = 0;
>      config->maildir_synchronize_flags = TRUE;
>      config->search_exclude_tags = NULL;
>      config->search_exclude_tags_length = 0;
> @@ -360,7 +367,11 @@ notmuch_config_open (void *ctx,
>          const char *tags[] = { "unread", "inbox" };
>  	notmuch_config_set_new_tags (config, tags, 2);
>      }
> -

Looks like you lost a blank line.

> +#if 0 /* No point setting empty list -- it's not written */
> +    if (notmuch_config_get_new_ignore (config, &tmp) == NULL) {
> +	notmuch_config_set_new_ignore (config, NULL, 0);
> +    }
> +#endif

This *does* matter.  It affects whether or not this key appears in
saved configuration files (both updated ones and new ones).  For
discoverability's sake (especially since we don't have a way to update
the section's comment), it should appear, even though it's blank.

(For the record, I think 'notmuch config set's behavior of removing a
key if given no arguments is broken; because of how we handle
defaults, this is completely asymmetric behavior, plus it seems to
screw up group comments.)

Also, there should be another blank line after the if.

>      if (notmuch_config_get_search_exclude_tags (config, &tmp) == NULL) {
>  	if (is_new) {
>  	    const char *tags[] = { "deleted", "spam" };
> @@ -609,6 +620,15 @@ notmuch_config_get_new_tags (notmuch_config_t *config,   size_t *length)
>  			     &(config->new_tags_length), length);
>  }
>  
> +const char **
> +notmuch_config_get_new_ignore (notmuch_config_t *config,   size_t *length)
> +{
> +    return _config_get_list (config, "new", "ignore",
> +			     &(config->new_ignore),
> +			     &(config->new_ignore_length), length);
> +}
> +
> +

One extra blank line.

>  void
>  notmuch_config_set_user_other_email (notmuch_config_t *config,
>  				     const char *list[],
> @@ -627,6 +647,17 @@ notmuch_config_set_new_tags (notmuch_config_t *config,
>  		     &(config->new_tags));
>  }
>  
> +#if 0 /* UNNEEDED SO FAR */

Include this, even if we don't need it.

> +void
> +notmuch_config_set_new_ignore (notmuch_config_t *config,
> +			       const char *list[],
> +			       size_t length)
> +{
> +    _config_set_list (config, "new", "ignore", list, length,
> +		     &(config->new_ignore));
> +}
> +#endif
> +
>  const char **
>  notmuch_config_get_search_exclude_tags (notmuch_config_t *config, size_t *length)
>  {
> diff --git a/notmuch-new.c b/notmuch-new.c
> index a569a54..36d5c5d 100644
> --- a/notmuch-new.c
> +++ b/notmuch-new.c
> @@ -39,6 +39,8 @@ typedef struct {
>      int verbose;
>      const char **new_tags;
>      size_t new_tags_length;
> +    const char **new_ignore;
> +    size_t new_ignore_length;
>  
>      int total_files;
>      int processed_files;
> @@ -181,6 +183,20 @@ _entries_resemble_maildir (struct dirent **entries, int count)
>      return 0;
>  }
>  
> +/* Check if user asked to ignore these files/directories */
> +

No blank line.

> +static int

Could be notmuch_bool_t (and TRUE and FALSE below).  Up to you.

> +_entry_in_ignore_list (const char *entry, add_files_state_t *state)
> +{
> +    size_t i, ignore_length = state->new_ignore_length;
> +
> +    for (i = 0; i < ignore_length; i++)
> +	if (strcmp (entry, state->new_ignore[i]) == 0)
> +	    return 1;
> +
> +    return 0;
> +}
> +
>  /* Examine 'path' recursively as follows:
>   *
>   *   o Ask the filesystem for the mtime of 'path' (fs_mtime)
> @@ -320,15 +336,15 @@ add_files_recursive (notmuch_database_t *notmuch,
>  	}
>  
>  	/* Ignore special directories to avoid infinite recursion.
> -	 * Also ignore the .notmuch directory and any "tmp" directory
> -	 * that appears within a maildir.
> +	 * Also ignore the .notmuch directory, any "tmp" directory
> +	 * that appears within a maildir and files/directories
> +	 * user have configured to be ignored.

s/user have/the user has/

>  	 */
> -	/* XXX: Eventually we'll want more sophistication to let the
> -	 * user specify files to be ignored. */
>  	if (strcmp (entry->d_name, ".") == 0 ||
>  	    strcmp (entry->d_name, "..") == 0 ||
>  	    (is_maildir && strcmp (entry->d_name, "tmp") == 0) ||
> -	    strcmp (entry->d_name, ".notmuch") ==0)
> +	    strcmp (entry->d_name, ".notmuch") == 0 ||
> +	    _entry_in_ignore_list (entry->d_name, state))

Ah, nice simplification.

>  	{
>  	    continue;
>  	}
> @@ -369,6 +385,10 @@ add_files_recursive (notmuch_database_t *notmuch,
>  
>          entry = fs_entries[i];
>  
> +	/* Ignore files & directories user has configured to be ignored */
> +	if (_entry_in_ignore_list (entry->d_name, state))
> +	    continue;
> +
>  	/* Check if we've walked past any names in db_files or
>  	 * db_subdirs. If so, these have been deleted. */
>  	while (notmuch_filenames_valid (db_files) &&
> @@ -648,7 +668,7 @@ add_files (notmuch_database_t *notmuch,
>   * initialized to zero by the top-level caller before calling
>   * count_files). */
>  static void
> -count_files (const char *path, int *count)
> +count_files (const char *path, int *count, add_files_state_t *state)
>  {
>      struct dirent *entry = NULL;
>      char *next;
> @@ -670,13 +690,13 @@ count_files (const char *path, int *count)
>          entry = fs_entries[i++];
>  
>  	/* Ignore special directories to avoid infinite recursion.
> -	 * Also ignore the .notmuch directory.
> +	 * Also ignore the .notmuch directory and files/directories
> +	 * user have configured to be ignored.
>  	 */
> -	/* XXX: Eventually we'll want more sophistication to let the
> -	 * user specify files to be ignored. */
>  	if (strcmp (entry->d_name, ".") == 0 ||
>  	    strcmp (entry->d_name, "..") == 0 ||
> -	    strcmp (entry->d_name, ".notmuch") == 0)
> +	    strcmp (entry->d_name, ".notmuch") == 0 ||
> +	    _entry_in_ignore_list (entry->d_name, state))
>  	{
>  	    continue;
>  	}
> @@ -697,7 +717,7 @@ count_files (const char *path, int *count)
>  		fflush (stdout);
>  	    }
>  	} else if (S_ISDIR (st.st_mode)) {
> -	    count_files (next, count);
> +	    count_files (next, count, state);
>  	}
>  
>  	free (next);
> @@ -837,6 +857,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
>  	return 1;
>  
>      add_files_state.new_tags = notmuch_config_get_new_tags (config, &add_files_state.new_tags_length);
> +    add_files_state.new_ignore = notmuch_config_get_new_ignore (config, &add_files_state.new_ignore_length);
>      add_files_state.synchronize_flags = notmuch_config_get_maildir_synchronize_flags (config);
>      db_path = notmuch_config_get_database_path (config);
>  
> @@ -852,7 +873,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
>  	int count;
>  
>  	count = 0;
> -	count_files (db_path, &count);
> +	count_files (db_path, &count, &add_files_state);
>  	if (interrupted)
>  	    return 1;
>  

Re: [PATCH] test: add tests wrt ignoring user-specified files and directories

[Tomi Ollila]

On Wed,  1 Feb 2012 15:12:53 +0100, Pieter Praet <pieter@praet.org> wrote:
> Files and directories which are specified in 'new.ignore' in the
> config file shouldn't be indexed nor reported by `notmuch new'.
> 
> ---

+1 

tested file order with:

$ touch ~/mail/mails/a
$ touch ~/mail/mails/d
$ touch ~/mail/mails/o
$ touch ~/mail/mails/p
$ touch ~/mail/mails/g
$ touch ~/mail/mails/z
$ touch ~/mail/mails/u
$ touch ~/mail/mails/l     

$ notmuch new
Note: Ignoring non-mail file: /home/too/mail/mails/a
Note: Ignoring non-mail file: /home/too/mail/mails/d
Note: Ignoring non-mail file: /home/too/mail/mails/g
Note: Ignoring non-mail file: /home/too/mail/mails/l
Note: Ignoring non-mail file: /home/too/mail/mails/o
Note: Ignoring non-mail file: /home/too/mail/mails/p
Note: Ignoring non-mail file: /home/too/mail/mails/u
Note: Ignoring non-mail file: /home/too/mail/mails/z

So we can be pretty confident that files are printed
in ASCII order.

Tomi

Re: [PATCH] test: add tests wrt ignoring user-specified files and directories

[Austin Clements]

On Fri, 03 Feb 2012 14:14:21 +0200, Tomi Ollila <tomi.ollila@iki.fi> wrote:
> On Wed,  1 Feb 2012 15:12:53 +0100, Pieter Praet <pieter@praet.org> wrote:
> > Files and directories which are specified in 'new.ignore' in the
> > config file shouldn't be indexed nor reported by `notmuch new'.
> > 
> > ---
> 
> +1 
> 
> tested file order with:
> 
> $ touch ~/mail/mails/a
> $ touch ~/mail/mails/d
> $ touch ~/mail/mails/o
> $ touch ~/mail/mails/p
> $ touch ~/mail/mails/g
> $ touch ~/mail/mails/z
> $ touch ~/mail/mails/u
> $ touch ~/mail/mails/l     
> 
> $ notmuch new
> Note: Ignoring non-mail file: /home/too/mail/mails/a
> Note: Ignoring non-mail file: /home/too/mail/mails/d
> Note: Ignoring non-mail file: /home/too/mail/mails/g
> Note: Ignoring non-mail file: /home/too/mail/mails/l
> Note: Ignoring non-mail file: /home/too/mail/mails/o
> Note: Ignoring non-mail file: /home/too/mail/mails/p
> Note: Ignoring non-mail file: /home/too/mail/mails/u
> Note: Ignoring non-mail file: /home/too/mail/mails/z
> 
> So we can be pretty confident that files are printed
> in ASCII order.

This is true if the containing directory has been scanned by notmuch
before (which is sufficiently true in this patch).  If the containing
directory is new, they're in inode order (so no guarantees).

Re: [PATCH] test: add tests wrt ignoring user-specified files and directories

[Austin Clements]

On Wed,  1 Feb 2012 15:12:53 +0100, Pieter Praet <pieter@praet.org> wrote:
> Files and directories which are specified in 'new.ignore' in the
> config file shouldn't be indexed nor reported by `notmuch new'.
> 
> ---
>  test/new |   23 +++++++++++++++++++++++
>  1 files changed, 23 insertions(+), 0 deletions(-)
> 
> diff --git a/test/new b/test/new
> index 49f390d..740ba05 100755
> --- a/test/new
> +++ b/test/new
> @@ -153,4 +153,27 @@ rm -rf "${MAIL_DIR}"/two
>  output=$(NOTMUCH_NEW)
>  test_expect_equal "$output" "No new mail. Removed 3 messages."
>  
> +test_begin_subtest "Skip and report non-mail files"
> +generate_message
> +mkdir -p "${MAIL_DIR}"/.git && touch "${MAIL_DIR}"/.git/config
> +touch "${MAIL_DIR}"/ignored_file
> +touch "${MAIL_DIR}"/.ignored_hidden_file
> +output=$(NOTMUCH_NEW 2>&1)
> +test_expect_equal "$output" \
> +"Note: Ignoring non-mail file: ${MAIL_DIR}/.git/config
> +Note: Ignoring non-mail file: ${MAIL_DIR}/.ignored_hidden_file
> +Note: Ignoring non-mail file: ${MAIL_DIR}/ignored_file
> +Added 1 new message to the database."
> +
> +test_begin_subtest "Ignore files and directories specified in new.ignore"
> +test_subtest_known_broken
> +generate_message
> +notmuch config set new.ignore .git ignored_file .ignored_hidden_file
> +mkdir -p "${MAIL_DIR}"/.git && touch "${MAIL_DIR}"/.git/config

This doesn't change .git's mtime, so notmuch new may not scan it (*may*
not because it's possible/likely notmuch refused to record the mtime in
the above test on account of lack of granularity).  Probably you should
just touch "${MAIL_DIR}"/.git.  Or maybe touch a new file under it.

> +touch "${MAIL_DIR}"/ignored_file
> +touch "${MAIL_DIR}"/.ignored_hidden_file

These aren't necessary since these files already exist and they won't
touch the directory's mtime (though the generate_message will).

> +output=$(NOTMUCH_NEW 2>&1)
> +test_expect_equal "$output" "Added 1 new message to the database."
> +
> +
>  test_done
> -- 
> 1.7.8.1
> 
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch
> 

[PATCH v6 1/3] test: add tests wrt ignoring user-specified files and directories

[Tomi Ollila]

Files and directories which are specified in 'new.ignore' in the
config file shouldn't be indexed nor reported by `notmuch new'.

This is basically Pieter's work with Austin's comments addressed.
---
 test/new |   22 ++++++++++++++++++++++
 1 files changed, 22 insertions(+), 0 deletions(-)

diff --git a/test/new b/test/new
index 49f390d..5ce8811 100755
--- a/test/new
+++ b/test/new
@@ -153,4 +153,26 @@ rm -rf "${MAIL_DIR}"/two
 output=$(NOTMUCH_NEW)
 test_expect_equal "$output" "No new mail. Removed 3 messages."
 
+# This test depends that notmuch new has been run at least once.
+test_begin_subtest "Skip and report non-mail files"
+generate_message
+mkdir -p "${MAIL_DIR}"/.git && touch "${MAIL_DIR}"/.git/config
+touch "${MAIL_DIR}"/ignored_file
+touch "${MAIL_DIR}"/.ignored_hidden_file
+output=$(NOTMUCH_NEW 2>&1)
+test_expect_equal "$output" \
+"Note: Ignoring non-mail file: ${MAIL_DIR}/.git/config
+Note: Ignoring non-mail file: ${MAIL_DIR}/.ignored_hidden_file
+Note: Ignoring non-mail file: ${MAIL_DIR}/ignored_file
+Added 1 new message to the database."
+
+test_begin_subtest "Ignore files and directories specified in new.ignore"
+test_subtest_known_broken
+generate_message
+notmuch config set new.ignore .git ignored_file .ignored_hidden_file
+touch "${MAIL_DIR}"/.git # change .git's mtime for notmuch new to rescan.
+output=$(NOTMUCH_NEW 2>&1)
+test_expect_equal "$output" "Added 1 new message to the database."
+
+
 test_done
-- 
1.7.6.5

[PATCH v6 2/3] add support for user-specified files & directories to ignore

[Tomi Ollila]

A new configuration key 'new.ignore' is used to determine which
files and directories user wants not to be scanned as new mails.

Mark the corresponding test as no longer broken.

This work merges my previous attempts and Andreas Amann's work
in id:"ylp7hi23mw8.fsf@tyndall.ie"
---
 notmuch-client.h |    9 +++++++++
 notmuch-config.c |   30 +++++++++++++++++++++++++++++-
 notmuch-new.c    |   45 +++++++++++++++++++++++++++++++++------------
 test/new         |    1 -
 4 files changed, 71 insertions(+), 14 deletions(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index e0eb594..f1762ae 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -250,6 +250,15 @@ notmuch_config_set_new_tags (notmuch_config_t *config,
 			     const char *new_tags[],
 			     size_t length);
 
+const char **
+notmuch_config_get_new_ignore (notmuch_config_t *config,
+			       size_t *length);
+
+void
+notmuch_config_set_new_ignore (notmuch_config_t *config,
+			       const char *new_ignore[],
+			       size_t length);
+
 notmuch_bool_t
 notmuch_config_get_maildir_synchronize_flags (notmuch_config_t *config);
 
diff --git a/notmuch-config.c b/notmuch-config.c
index a124e34..1f01128 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -44,7 +44,10 @@ static const char new_config_comment[] =
     " The following options are supported here:\n"
     "\n"
     "\ttags	A list (separated by ';') of the tags that will be\n"
-    "\t	added to all messages incorporated by \"notmuch new\".\n";
+    "\t	added to all messages incorporated by \"notmuch new\".\n"
+    "\n"
+    "\tignore	A list (separated by ';') of file and directory names\n"
+    "\t	that will not be searched for messages by \"notmuch new\".\n";
 
 static const char user_config_comment[] =
     " User configuration\n"
@@ -105,6 +108,8 @@ struct _notmuch_config {
     size_t user_other_email_length;
     const char **new_tags;
     size_t new_tags_length;
+    const char **new_ignore;
+    size_t new_ignore_length;
     notmuch_bool_t maildir_synchronize_flags;
     const char **search_exclude_tags;
     size_t search_exclude_tags_length;
@@ -264,6 +269,8 @@ notmuch_config_open (void *ctx,
     config->user_other_email_length = 0;
     config->new_tags = NULL;
     config->new_tags_length = 0;
+    config->new_ignore = NULL;
+    config->new_ignore_length = 0;
     config->maildir_synchronize_flags = TRUE;
     config->search_exclude_tags = NULL;
     config->search_exclude_tags_length = 0;
@@ -361,6 +368,10 @@ notmuch_config_open (void *ctx,
 	notmuch_config_set_new_tags (config, tags, 2);
     }
 
+    if (notmuch_config_get_new_ignore (config, &tmp) == NULL) {
+	notmuch_config_set_new_ignore (config, NULL, 0);
+    }
+
     if (notmuch_config_get_search_exclude_tags (config, &tmp) == NULL) {
 	if (is_new) {
 	    const char *tags[] = { "deleted", "spam" };
@@ -609,6 +620,14 @@ notmuch_config_get_new_tags (notmuch_config_t *config,   size_t *length)
 			     &(config->new_tags_length), length);
 }
 
+const char **
+notmuch_config_get_new_ignore (notmuch_config_t *config, size_t *length)
+{
+    return _config_get_list (config, "new", "ignore",
+			     &(config->new_ignore),
+			     &(config->new_ignore_length), length);
+}
+
 void
 notmuch_config_set_user_other_email (notmuch_config_t *config,
 				     const char *list[],
@@ -627,6 +646,15 @@ notmuch_config_set_new_tags (notmuch_config_t *config,
 		     &(config->new_tags));
 }
 
+void
+notmuch_config_set_new_ignore (notmuch_config_t *config,
+			       const char *list[],
+			       size_t length)
+{
+    _config_set_list (config, "new", "ignore", list, length,
+		     &(config->new_ignore));
+}
+
 const char **
 notmuch_config_get_search_exclude_tags (notmuch_config_t *config, size_t *length)
 {
diff --git a/notmuch-new.c b/notmuch-new.c
index a569a54..8a615e6 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -39,6 +39,8 @@ typedef struct {
     int verbose;
     const char **new_tags;
     size_t new_tags_length;
+    const char **new_ignore;
+    size_t new_ignore_length;
 
     int total_files;
     int processed_files;
@@ -181,6 +183,20 @@ _entries_resemble_maildir (struct dirent **entries, int count)
     return 0;
 }
 
+/* Test if the file/directory is to be ignored.
+ */
+static notmuch_bool_t
+_entry_in_ignore_list (const char *entry, add_files_state_t *state)
+{
+    size_t i;
+
+    for (i = 0; i < state->new_ignore_length; i++)
+	if (strcmp (entry, state->new_ignore[i]) == 0)
+	    return TRUE;
+
+    return FALSE;
+}
+
 /* Examine 'path' recursively as follows:
  *
  *   o Ask the filesystem for the mtime of 'path' (fs_mtime)
@@ -320,15 +336,15 @@ add_files_recursive (notmuch_database_t *notmuch,
 	}
 
 	/* Ignore special directories to avoid infinite recursion.
-	 * Also ignore the .notmuch directory and any "tmp" directory
-	 * that appears within a maildir.
+	 * Also ignore the .notmuch directory, any "tmp" directory
+	 * that appears within a maildir and files/directories
+	 * the user has configured to be ignored.
 	 */
-	/* XXX: Eventually we'll want more sophistication to let the
-	 * user specify files to be ignored. */
 	if (strcmp (entry->d_name, ".") == 0 ||
 	    strcmp (entry->d_name, "..") == 0 ||
 	    (is_maildir && strcmp (entry->d_name, "tmp") == 0) ||
-	    strcmp (entry->d_name, ".notmuch") ==0)
+	    strcmp (entry->d_name, ".notmuch") == 0 ||
+	    _entry_in_ignore_list (entry->d_name, state))
 	{
 	    continue;
 	}
@@ -369,6 +385,10 @@ add_files_recursive (notmuch_database_t *notmuch,
 
         entry = fs_entries[i];
 
+	/* Ignore files & directories user has configured to be ignored */
+	if (_entry_in_ignore_list (entry->d_name, state))
+	    continue;
+
 	/* Check if we've walked past any names in db_files or
 	 * db_subdirs. If so, these have been deleted. */
 	while (notmuch_filenames_valid (db_files) &&
@@ -648,7 +668,7 @@ add_files (notmuch_database_t *notmuch,
  * initialized to zero by the top-level caller before calling
  * count_files). */
 static void
-count_files (const char *path, int *count)
+count_files (const char *path, int *count, add_files_state_t *state)
 {
     struct dirent *entry = NULL;
     char *next;
@@ -670,13 +690,13 @@ count_files (const char *path, int *count)
         entry = fs_entries[i++];
 
 	/* Ignore special directories to avoid infinite recursion.
-	 * Also ignore the .notmuch directory.
+	 * Also ignore the .notmuch directory and files/directories
+	 * the user has configured to be ignored.
 	 */
-	/* XXX: Eventually we'll want more sophistication to let the
-	 * user specify files to be ignored. */
 	if (strcmp (entry->d_name, ".") == 0 ||
 	    strcmp (entry->d_name, "..") == 0 ||
-	    strcmp (entry->d_name, ".notmuch") == 0)
+	    strcmp (entry->d_name, ".notmuch") == 0 ||
+	    _entry_in_ignore_list (entry->d_name, state))
 	{
 	    continue;
 	}
@@ -697,7 +717,7 @@ count_files (const char *path, int *count)
 		fflush (stdout);
 	    }
 	} else if (S_ISDIR (st.st_mode)) {
-	    count_files (next, count);
+	    count_files (next, count, state);
 	}
 
 	free (next);
@@ -837,6 +857,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
 	return 1;
 
     add_files_state.new_tags = notmuch_config_get_new_tags (config, &add_files_state.new_tags_length);
+    add_files_state.new_ignore = notmuch_config_get_new_ignore (config, &add_files_state.new_ignore_length);
     add_files_state.synchronize_flags = notmuch_config_get_maildir_synchronize_flags (config);
     db_path = notmuch_config_get_database_path (config);
 
@@ -852,7 +873,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
 	int count;
 
 	count = 0;
-	count_files (db_path, &count);
+	count_files (db_path, &count, &add_files_state);
 	if (interrupted)
 	    return 1;
 
diff --git a/test/new b/test/new
index 5ce8811..2af63a1 100755
--- a/test/new
+++ b/test/new
@@ -167,7 +167,6 @@ Note: Ignoring non-mail file: ${MAIL_DIR}/ignored_file
 Added 1 new message to the database."
 
 test_begin_subtest "Ignore files and directories specified in new.ignore"
-test_subtest_known_broken
 generate_message
 notmuch config set new.ignore .git ignored_file .ignored_hidden_file
 touch "${MAIL_DIR}"/.git # change .git's mtime for notmuch new to rescan.
-- 
1.7.6.5

[PATCH v6 3/3] NEWS: add news section for new.ignore

[Tomi Ollila]

Added NEWS section 'Mail store folder/file ignore'.
---
 NEWS |    6 ++++++
 1 files changed, 6 insertions(+), 0 deletions(-)

diff --git a/NEWS b/NEWS
index 5c5b645..59da584 100644
--- a/NEWS
+++ b/NEWS
@@ -29,6 +29,12 @@ Tag exclusion
 
     notmuch config set search.exclude_tags deleted spam
 
+Mail store folder/file ignore
+
+  New configuration option `new.ignore` is used to inform notmuch about
+  folders and files that are not mail files or contain material that is
+  not desired to be known by notmuch.
+
 Emacs Interface
 ---------------
 
-- 
1.7.6.5

Re: [PATCH v6 2/3] add support for user-specified files & directories to ignore

[Tomi Ollila]

On Mon,  6 Feb 2012 11:28:24 +0200, Tomi Ollila <tomi.ollila@iki.fi> wrote:
> A new configuration key 'new.ignore' is used to determine which
> files and directories user wants not to be scanned as new mails.
> 
> Mark the corresponding test as no longer broken.
> 
> This work merges my previous attempts and Andreas Amann's work
> in id:"ylp7hi23mw8.fsf@tyndall.ie"

I marked this patch with 'needs-review' for consistency with
the 2 other patches. No code changes to the previous version.

Tomi

Re: [PATCH v6 1/3] test: add tests wrt ignoring user-specified files and directories

[Tomi Ollila]

Those interested in this patch set please check lates changes.
The test patch (1/3) has been slighty modified from Pieter's
version and NEWS patch (3/3) is new. The actual functionality
in patch 2/3 is exactly the same as in previous version.

This applies cleanly in top of current master
( a5674c21584a32a4f8a8a3e9ea3c3576f772e744 ) pushed Sun 2012-02-12.

Tomi

Re: [PATCH v6 1/3] test: add tests wrt ignoring user-specified files and directories

[Austin Clements]

On Mon,  6 Feb 2012 11:28:23 +0200, Tomi Ollila <tomi.ollila@iki.fi> wrote:
> Files and directories which are specified in 'new.ignore' in the
> config file shouldn't be indexed nor reported by `notmuch new'.
> 
> This is basically Pieter's work with Austin's comments addressed.
> ---
>  test/new |   22 ++++++++++++++++++++++
>  1 files changed, 22 insertions(+), 0 deletions(-)
> 
> diff --git a/test/new b/test/new
> index 49f390d..5ce8811 100755
> --- a/test/new
> +++ b/test/new
> @@ -153,4 +153,26 @@ rm -rf "${MAIL_DIR}"/two
>  output=$(NOTMUCH_NEW)
>  test_expect_equal "$output" "No new mail. Removed 3 messages."
>  
> +# This test depends that notmuch new has been run at least once.

s/depends/requires/

> +test_begin_subtest "Skip and report non-mail files"
> +generate_message
> +mkdir -p "${MAIL_DIR}"/.git && touch "${MAIL_DIR}"/.git/config
> +touch "${MAIL_DIR}"/ignored_file
> +touch "${MAIL_DIR}"/.ignored_hidden_file
> +output=$(NOTMUCH_NEW 2>&1)
> +test_expect_equal "$output" \
> +"Note: Ignoring non-mail file: ${MAIL_DIR}/.git/config
> +Note: Ignoring non-mail file: ${MAIL_DIR}/.ignored_hidden_file
> +Note: Ignoring non-mail file: ${MAIL_DIR}/ignored_file
> +Added 1 new message to the database."
> +
> +test_begin_subtest "Ignore files and directories specified in new.ignore"
> +test_subtest_known_broken
> +generate_message
> +notmuch config set new.ignore .git ignored_file .ignored_hidden_file
> +touch "${MAIL_DIR}"/.git # change .git's mtime for notmuch new to rescan.
> +output=$(NOTMUCH_NEW 2>&1)
> +test_expect_equal "$output" "Added 1 new message to the database."
> +
> +
>  test_done
> -- 
> 1.7.6.5
> 
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch
> 

Re: [PATCH v6 3/3] NEWS: add news section for new.ignore

[Austin Clements]

On Mon,  6 Feb 2012 11:28:25 +0200, Tomi Ollila <tomi.ollila@iki.fi> wrote:
> Added NEWS section 'Mail store folder/file ignore'.
> ---
>  NEWS |    6 ++++++
>  1 files changed, 6 insertions(+), 0 deletions(-)
> 
> diff --git a/NEWS b/NEWS
> index 5c5b645..59da584 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -29,6 +29,12 @@ Tag exclusion
>  
>      notmuch config set search.exclude_tags deleted spam
>  
> +Mail store folder/file ignore
> +
> +  New configuration option `new.ignore` is used to inform notmuch about
> +  folders and files that are not mail files or contain material that is
> +  not desired to be known by notmuch.
> +

This is a bit unwieldy.  How about something based on the config file
comment?

  A new configuration option, `new.ignore`, lets users specify a
  ;-separated list of file and directory names that will not be searched
  for messages by "notmuch new".

Plus, this way it's not so bad that we can't upgrade an existing config
file comment.

>  Emacs Interface
>  ---------------
>  
> -- 
> 1.7.6.5
> 
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch
> 

Re: [PATCH v6 1/3] test: add tests wrt ignoring user-specified files and directories

[Austin Clements]

On Mon, 13 Feb 2012 10:20:28 +0200, Tomi Ollila <tomi.ollila@iki.fi> wrote:
> Those interested in this patch set please check lates changes.
> The test patch (1/3) has been slighty modified from Pieter's
> version and NEWS patch (3/3) is new. The actual functionality
> in patch 2/3 is exactly the same as in previous version.
> 
> This applies cleanly in top of current master
> ( a5674c21584a32a4f8a8a3e9ea3c3576f772e744 ) pushed Sun 2012-02-12.

This series LGTM other than the two minor wording changes I just sent
out.

[PATCH v7 0/3] NEWS and test comment adjustments

[Tomi Ollila]

This version of 'new.ignore' patch series addresses Austin's last
comments in NEWS and test/new documentation parts.

No functional changes.

[PATCH v7 1/3] test: add tests wrt ignoring user-specified files and directories

[Tomi Ollila]

Files and directories which are specified in 'new.ignore' in the
config file shouldn't be indexed nor reported by `notmuch new'.

This is basically Pieter's work with Austin's comments addressed.
---
 test/new |   22 ++++++++++++++++++++++
 1 files changed, 22 insertions(+), 0 deletions(-)

diff --git a/test/new b/test/new
index 49f390d..e453684 100755
--- a/test/new
+++ b/test/new
@@ -153,4 +153,26 @@ rm -rf "${MAIL_DIR}"/two
 output=$(NOTMUCH_NEW)
 test_expect_equal "$output" "No new mail. Removed 3 messages."
 
+# This test requires that notmuch new has been run at least once.
+test_begin_subtest "Skip and report non-mail files"
+generate_message
+mkdir -p "${MAIL_DIR}"/.git && touch "${MAIL_DIR}"/.git/config
+touch "${MAIL_DIR}"/ignored_file
+touch "${MAIL_DIR}"/.ignored_hidden_file
+output=$(NOTMUCH_NEW 2>&1)
+test_expect_equal "$output" \
+"Note: Ignoring non-mail file: ${MAIL_DIR}/.git/config
+Note: Ignoring non-mail file: ${MAIL_DIR}/.ignored_hidden_file
+Note: Ignoring non-mail file: ${MAIL_DIR}/ignored_file
+Added 1 new message to the database."
+
+test_begin_subtest "Ignore files and directories specified in new.ignore"
+test_subtest_known_broken
+generate_message
+notmuch config set new.ignore .git ignored_file .ignored_hidden_file
+touch "${MAIL_DIR}"/.git # change .git's mtime for notmuch new to rescan.
+output=$(NOTMUCH_NEW 2>&1)
+test_expect_equal "$output" "Added 1 new message to the database."
+
+
 test_done
-- 
1.7.8.2

[PATCH v7 2/3] add support for user-specified files & directories to ignore

[Tomi Ollila]

A new configuration key 'new.ignore' is used to determine which
files and directories user wants not to be scanned as new mails.

Mark the corresponding test as no longer broken.
This work merges my previous attempts and Andreas Amann's work
in id:"ylp7hi23mw8.fsf@tyndall.ie"
---
 notmuch-client.h |    9 +++++++++
 notmuch-config.c |   30 +++++++++++++++++++++++++++++-
 notmuch-new.c    |   45 +++++++++++++++++++++++++++++++++------------
 test/new         |    1 -
 4 files changed, 71 insertions(+), 14 deletions(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index 60828aa..f4a62cc 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -250,6 +250,15 @@ notmuch_config_set_new_tags (notmuch_config_t *config,
 			     const char *new_tags[],
 			     size_t length);
 
+const char **
+notmuch_config_get_new_ignore (notmuch_config_t *config,
+			       size_t *length);
+
+void
+notmuch_config_set_new_ignore (notmuch_config_t *config,
+			       const char *new_ignore[],
+			       size_t length);
+
 notmuch_bool_t
 notmuch_config_get_maildir_synchronize_flags (notmuch_config_t *config);
 
diff --git a/notmuch-config.c b/notmuch-config.c
index a124e34..1f01128 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -44,7 +44,10 @@ static const char new_config_comment[] =
     " The following options are supported here:\n"
     "\n"
     "\ttags	A list (separated by ';') of the tags that will be\n"
-    "\t	added to all messages incorporated by \"notmuch new\".\n";
+    "\t	added to all messages incorporated by \"notmuch new\".\n"
+    "\n"
+    "\tignore	A list (separated by ';') of file and directory names\n"
+    "\t	that will not be searched for messages by \"notmuch new\".\n";
 
 static const char user_config_comment[] =
     " User configuration\n"
@@ -105,6 +108,8 @@ struct _notmuch_config {
     size_t user_other_email_length;
     const char **new_tags;
     size_t new_tags_length;
+    const char **new_ignore;
+    size_t new_ignore_length;
     notmuch_bool_t maildir_synchronize_flags;
     const char **search_exclude_tags;
     size_t search_exclude_tags_length;
@@ -264,6 +269,8 @@ notmuch_config_open (void *ctx,
     config->user_other_email_length = 0;
     config->new_tags = NULL;
     config->new_tags_length = 0;
+    config->new_ignore = NULL;
+    config->new_ignore_length = 0;
     config->maildir_synchronize_flags = TRUE;
     config->search_exclude_tags = NULL;
     config->search_exclude_tags_length = 0;
@@ -361,6 +368,10 @@ notmuch_config_open (void *ctx,
 	notmuch_config_set_new_tags (config, tags, 2);
     }
 
+    if (notmuch_config_get_new_ignore (config, &tmp) == NULL) {
+	notmuch_config_set_new_ignore (config, NULL, 0);
+    }
+
     if (notmuch_config_get_search_exclude_tags (config, &tmp) == NULL) {
 	if (is_new) {
 	    const char *tags[] = { "deleted", "spam" };
@@ -609,6 +620,14 @@ notmuch_config_get_new_tags (notmuch_config_t *config,   size_t *length)
 			     &(config->new_tags_length), length);
 }
 
+const char **
+notmuch_config_get_new_ignore (notmuch_config_t *config, size_t *length)
+{
+    return _config_get_list (config, "new", "ignore",
+			     &(config->new_ignore),
+			     &(config->new_ignore_length), length);
+}
+
 void
 notmuch_config_set_user_other_email (notmuch_config_t *config,
 				     const char *list[],
@@ -627,6 +646,15 @@ notmuch_config_set_new_tags (notmuch_config_t *config,
 		     &(config->new_tags));
 }
 
+void
+notmuch_config_set_new_ignore (notmuch_config_t *config,
+			       const char *list[],
+			       size_t length)
+{
+    _config_set_list (config, "new", "ignore", list, length,
+		     &(config->new_ignore));
+}
+
 const char **
 notmuch_config_get_search_exclude_tags (notmuch_config_t *config, size_t *length)
 {
diff --git a/notmuch-new.c b/notmuch-new.c
index 8dbebb3..4f13535 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -39,6 +39,8 @@ typedef struct {
     int verbose;
     const char **new_tags;
     size_t new_tags_length;
+    const char **new_ignore;
+    size_t new_ignore_length;
 
     int total_files;
     int processed_files;
@@ -181,6 +183,20 @@ _entries_resemble_maildir (struct dirent **entries, int count)
     return 0;
 }
 
+/* Test if the file/directory is to be ignored.
+ */
+static notmuch_bool_t
+_entry_in_ignore_list (const char *entry, add_files_state_t *state)
+{
+    size_t i;
+
+    for (i = 0; i < state->new_ignore_length; i++)
+	if (strcmp (entry, state->new_ignore[i]) == 0)
+	    return TRUE;
+
+    return FALSE;
+}
+
 /* Examine 'path' recursively as follows:
  *
  *   o Ask the filesystem for the mtime of 'path' (fs_mtime)
@@ -320,15 +336,15 @@ add_files_recursive (notmuch_database_t *notmuch,
 	}
 
 	/* Ignore special directories to avoid infinite recursion.
-	 * Also ignore the .notmuch directory and any "tmp" directory
-	 * that appears within a maildir.
+	 * Also ignore the .notmuch directory, any "tmp" directory
+	 * that appears within a maildir and files/directories
+	 * the user has configured to be ignored.
 	 */
-	/* XXX: Eventually we'll want more sophistication to let the
-	 * user specify files to be ignored. */
 	if (strcmp (entry->d_name, ".") == 0 ||
 	    strcmp (entry->d_name, "..") == 0 ||
 	    (is_maildir && strcmp (entry->d_name, "tmp") == 0) ||
-	    strcmp (entry->d_name, ".notmuch") ==0)
+	    strcmp (entry->d_name, ".notmuch") == 0 ||
+	    _entry_in_ignore_list (entry->d_name, state))
 	{
 	    continue;
 	}
@@ -369,6 +385,10 @@ add_files_recursive (notmuch_database_t *notmuch,
 
         entry = fs_entries[i];
 
+	/* Ignore files & directories user has configured to be ignored */
+	if (_entry_in_ignore_list (entry->d_name, state))
+	    continue;
+
 	/* Check if we've walked past any names in db_files or
 	 * db_subdirs. If so, these have been deleted. */
 	while (notmuch_filenames_valid (db_files) &&
@@ -650,7 +670,7 @@ add_files (notmuch_database_t *notmuch,
  * initialized to zero by the top-level caller before calling
  * count_files). */
 static void
-count_files (const char *path, int *count)
+count_files (const char *path, int *count, add_files_state_t *state)
 {
     struct dirent *entry = NULL;
     char *next;
@@ -672,13 +692,13 @@ count_files (const char *path, int *count)
         entry = fs_entries[i++];
 
 	/* Ignore special directories to avoid infinite recursion.
-	 * Also ignore the .notmuch directory.
+	 * Also ignore the .notmuch directory and files/directories
+	 * the user has configured to be ignored.
 	 */
-	/* XXX: Eventually we'll want more sophistication to let the
-	 * user specify files to be ignored. */
 	if (strcmp (entry->d_name, ".") == 0 ||
 	    strcmp (entry->d_name, "..") == 0 ||
-	    strcmp (entry->d_name, ".notmuch") == 0)
+	    strcmp (entry->d_name, ".notmuch") == 0 ||
+	    _entry_in_ignore_list (entry->d_name, state))
 	{
 	    continue;
 	}
@@ -699,7 +719,7 @@ count_files (const char *path, int *count)
 		fflush (stdout);
 	    }
 	} else if (S_ISDIR (st.st_mode)) {
-	    count_files (next, count);
+	    count_files (next, count, state);
 	}
 
 	free (next);
@@ -841,6 +861,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
 	return 1;
 
     add_files_state.new_tags = notmuch_config_get_new_tags (config, &add_files_state.new_tags_length);
+    add_files_state.new_ignore = notmuch_config_get_new_ignore (config, &add_files_state.new_ignore_length);
     add_files_state.synchronize_flags = notmuch_config_get_maildir_synchronize_flags (config);
     db_path = notmuch_config_get_database_path (config);
 
@@ -856,7 +877,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
 	int count;
 
 	count = 0;
-	count_files (db_path, &count);
+	count_files (db_path, &count, &add_files_state);
 	if (interrupted)
 	    return 1;
 
diff --git a/test/new b/test/new
index e453684..79a6c97 100755
--- a/test/new
+++ b/test/new
@@ -167,7 +167,6 @@ Note: Ignoring non-mail file: ${MAIL_DIR}/ignored_file
 Added 1 new message to the database."
 
 test_begin_subtest "Ignore files and directories specified in new.ignore"
-test_subtest_known_broken
 generate_message
 notmuch config set new.ignore .git ignored_file .ignored_hidden_file
 touch "${MAIL_DIR}"/.git # change .git's mtime for notmuch new to rescan.
-- 
1.7.8.2

[PATCH v7 3/3] NEWS: add news section for new.ignore

[Tomi Ollila]

Added NEWS section 'Mail store folder/file ignore'.
---
 NEWS |    6 ++++++
 1 files changed, 6 insertions(+), 0 deletions(-)

diff --git a/NEWS b/NEWS
index f449fba..e9abb86 100644
--- a/NEWS
+++ b/NEWS
@@ -29,6 +29,12 @@ Tag exclusion
 
     notmuch config set search.exclude_tags deleted spam
 
+Mail store folder/file ignore
+
+   A new configuration option, `new.ignore`, lets users specify a
+   ;-separated list of file and directory names that will not be
+   searched for messages by "notmuch new".
+
 Emacs Interface
 ---------------
 
-- 
1.7.8.2

Re: [PATCH v7 0/3] NEWS and test comment adjustments

[David Bremner]

On Wed, 15 Feb 2012 11:17:29 +0200, Tomi Ollila <tomi.ollila@iki.fi> wrote:
> This version of 'new.ignore' patch series addresses Austin's last
> comments in NEWS and test/new documentation parts.
> 

Pushed, 

d

Re: [PATCH] test: add tests wrt ignoring user-specified files and directories

[Pieter Praet]

On Fri, 03 Feb 2012 17:44:55 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> On Wed,  1 Feb 2012 15:12:53 +0100, Pieter Praet <pieter@praet.org> wrote:
> > Files and directories which are specified in 'new.ignore' in the
> > config file shouldn't be indexed nor reported by `notmuch new'.
> > 
> > ---
> >  test/new |   23 +++++++++++++++++++++++
> >  1 files changed, 23 insertions(+), 0 deletions(-)
> > 
> > diff --git a/test/new b/test/new
> > index 49f390d..740ba05 100755
> > --- a/test/new
> > +++ b/test/new
> > @@ -153,4 +153,27 @@ rm -rf "${MAIL_DIR}"/two
> >  output=$(NOTMUCH_NEW)
> >  test_expect_equal "$output" "No new mail. Removed 3 messages."
> >  
> > +test_begin_subtest "Skip and report non-mail files"
> > +generate_message
> > +mkdir -p "${MAIL_DIR}"/.git && touch "${MAIL_DIR}"/.git/config
> > +touch "${MAIL_DIR}"/ignored_file
> > +touch "${MAIL_DIR}"/.ignored_hidden_file
> > +output=$(NOTMUCH_NEW 2>&1)
> > +test_expect_equal "$output" \
> > +"Note: Ignoring non-mail file: ${MAIL_DIR}/.git/config
> > +Note: Ignoring non-mail file: ${MAIL_DIR}/.ignored_hidden_file
> > +Note: Ignoring non-mail file: ${MAIL_DIR}/ignored_file
> > +Added 1 new message to the database."
> > +
> > +test_begin_subtest "Ignore files and directories specified in new.ignore"
> > +test_subtest_known_broken
> > +generate_message
> > +notmuch config set new.ignore .git ignored_file .ignored_hidden_file
> > +mkdir -p "${MAIL_DIR}"/.git && touch "${MAIL_DIR}"/.git/config
> 
> This doesn't change .git's mtime, so notmuch new may not scan it (*may*
> not because it's possible/likely notmuch refused to record the mtime in
> the above test on account of lack of granularity).  Probably you should
> just touch "${MAIL_DIR}"/.git.  Or maybe touch a new file under it.
>

Good point.


> > +touch "${MAIL_DIR}"/ignored_file
> > +touch "${MAIL_DIR}"/.ignored_hidden_file
> 
> These aren't necessary since these files already exist and they won't
> touch the directory's mtime (though the generate_message will).
>

Those are/were intended to keep the test self-contained (i.e.
independent of the previous one).  Probably should've added a
comment about that.


> > +output=$(NOTMUCH_NEW 2>&1)
> > +test_expect_equal "$output" "Added 1 new message to the database."
> > +
> > +
> >  test_done
> > -- 
> > 1.7.8.1
> > 
> > _______________________________________________
> > notmuch mailing list
> > notmuch@notmuchmail.org
> > http://notmuchmail.org/mailman/listinfo/notmuch
> > 

Thanks for reviewing!

Peace


-- 
Pieter

Re: [PATCH v7 0/3] NEWS and test comment adjustments

[Pieter Praet]

Very happy to see this go into mainline!  Thanks all!

We should make it very clear, though, that *every* file/directory
that matches one of the values in 'new.ignore' will be ignored.

Patches follow.


Peace

[PATCH 1/6] cli: update 'new.ignore' config file comment wrt file/directory matching

[Pieter Praet]

---
 notmuch-config.c |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/notmuch-config.c b/notmuch-config.c
index 1f01128..e9b2750 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -47,7 +47,10 @@ static const char new_config_comment[] =
     "\t	added to all messages incorporated by \"notmuch new\".\n"
     "\n"
     "\tignore	A list (separated by ';') of file and directory names\n"
-    "\t	that will not be searched for messages by \"notmuch new\".\n";
+    "\t	that will not be searched for messages by \"notmuch new\".\n"
+    "\n"
+    "\t	NOTE: *Every* file/directory that goes by one of those names will\n"
+    "\t	be ignored, independent of its depth/location in the mail store.\n";
 
 static const char user_config_comment[] =
     " User configuration\n"
-- 
1.7.8.1

[PATCH 2/6] NEWS: sync 'new.ignore' entry with its comment in notmuch-config.c

[Pieter Praet]

See previous commit.
---
 NEWS |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)

diff --git a/NEWS b/NEWS
index e9abb86..833ab5f 100644
--- a/NEWS
+++ b/NEWS
@@ -35,6 +35,9 @@ Mail store folder/file ignore
    ;-separated list of file and directory names that will not be
    searched for messages by "notmuch new".
 
+   NOTE: *Every* file/directory that goes by one of those names will
+   be ignored, independent of its depth/location in the mail store.
+
 Emacs Interface
 ---------------
 
-- 
1.7.8.1

[PATCH 3/6] cli: add '--debug' option to 'notmuch new'

[Pieter Praet]

This will be used in later patches to test the 'new.ignore'
config option more thoroughly.
---
 notmuch-new.c |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/notmuch-new.c b/notmuch-new.c
index 4f13535..ca12003 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -37,6 +37,7 @@ typedef struct _filename_list {
 typedef struct {
     int output_is_a_tty;
     int verbose;
+    int debug;
     const char **new_tags;
     size_t new_tags_length;
     const char **new_ignore;
@@ -842,6 +843,7 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
     notmuch_bool_t run_hooks = TRUE;
 
     add_files_state.verbose = 0;
+    add_files_state.debug = 0;
     add_files_state.output_is_a_tty = isatty (fileno (stdout));
 
     argc--; argv++; /* skip subcommand argument */
@@ -849,6 +851,8 @@ notmuch_new_command (void *ctx, int argc, char *argv[])
     for (i = 0; i < argc && argv[i][0] == '-'; i++) {
 	if (STRNCMP_LITERAL (argv[i], "--verbose") == 0) {
 	    add_files_state.verbose = 1;
+	} else if (strcmp (argv[i], "--debug") == 0) {
+	    add_files_state.debug = 1;
 	} else if (strcmp (argv[i], "--no-hooks") == 0) {
 	    run_hooks = FALSE;
 	} else {
-- 
1.7.8.1

[PATCH 4/6] cli: notmuch new: optionally output debug information when ignoring files/directories

[Pieter Praet]

When running 'notmuch new' with the '--debug' option, output debug
information regarding explicitly ignored files and directories.
---
 notmuch-new.c |   15 ++++++++++++++-
 1 files changed, 14 insertions(+), 1 deletions(-)

diff --git a/notmuch-new.c b/notmuch-new.c
index ca12003..6f34da1 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -347,6 +347,10 @@ add_files_recursive (notmuch_database_t *notmuch,
 	    strcmp (entry->d_name, ".notmuch") == 0 ||
 	    _entry_in_ignore_list (entry->d_name, state))
 	{
+	    if (_entry_in_ignore_list (entry->d_name, state) && state->debug)
+		printf ("(D) add_files_recursive, pass 1: explicitly ignoring %s/%s\n",
+			path,
+			entry->d_name);
 	    continue;
 	}
 
@@ -387,8 +391,13 @@ add_files_recursive (notmuch_database_t *notmuch,
         entry = fs_entries[i];
 
 	/* Ignore files & directories user has configured to be ignored */
-	if (_entry_in_ignore_list (entry->d_name, state))
+	if (_entry_in_ignore_list (entry->d_name, state)) {
+	    if (state->debug)
+		printf ("(D) add_files_recursive, pass 2: explicitly ignoring %s/%s\n",
+			path,
+			entry->d_name);
 	    continue;
+	}
 
 	/* Check if we've walked past any names in db_files or
 	 * db_subdirs. If so, these have been deleted. */
@@ -701,6 +710,10 @@ count_files (const char *path, int *count, add_files_state_t *state)
 	    strcmp (entry->d_name, ".notmuch") == 0 ||
 	    _entry_in_ignore_list (entry->d_name, state))
 	{
+	    if (_entry_in_ignore_list (entry->d_name, state) && state->debug)
+		printf ("(D) count_files: explicitly ignoring %s/%s\n",
+			path,
+			entry->d_name);
 	    continue;
 	}
 
-- 
1.7.8.1

[PATCH 5/6] test-lib.sh: pass 'NOTMUCH_NEW's args down to 'notmuch new'

[Pieter Praet]

Obviates the need to create a 'NOTMUCH_NEW' clone which runs
'notmuch new --debug'.  This will be used in a later patch.

Doesn't cause any issues for other tests.
---
 test/test-lib.sh |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/test/test-lib.sh b/test/test-lib.sh
index 0174e93..ade7000 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -534,7 +534,7 @@ test_emacs_expect_t () {
 
 NOTMUCH_NEW ()
 {
-    notmuch new | grep -v -E -e '^Processed [0-9]*( total)? file|Found [0-9]* total file'
+    notmuch new "${@}" | grep -v -E -e '^Processed [0-9]*( total)? file|Found [0-9]* total file'
 }
 
 notmuch_search_sanitize ()
-- 
1.7.8.1

[PATCH 6/6] test: another test wrt ignoring user-specified files and directories

[Pieter Praet]

Demonstrates that *every* file/directory which matches one of the values
in 'new.ignore' will be ignored, independent of its depth/location in
the mail store.
---
 test/new |   19 +++++++++++++++++++
 1 files changed, 19 insertions(+), 0 deletions(-)

diff --git a/test/new b/test/new
index 79a6c97..a7b7330 100755
--- a/test/new
+++ b/test/new
@@ -173,5 +173,24 @@ touch "${MAIL_DIR}"/.git # change .git's mtime for notmuch new to rescan.
 output=$(NOTMUCH_NEW 2>&1)
 test_expect_equal "$output" "Added 1 new message to the database."
 
+test_begin_subtest "Ignore files and directories specified in new.ignore (multiple occurrences)"
+notmuch config set new.ignore .git ignored_file .ignored_hidden_file
+touch "${MAIL_DIR}"/.git # change .git's mtime for notmuch new to rescan.
+mkdir -p "${MAIL_DIR}"/one/two/three/.git
+notmuch new > /dev/null # ensure that files/folders will be printed in ASCII order.
+touch "${MAIL_DIR}"/{one,one/two,one/two/three}/ignored_file
+output=$(NOTMUCH_NEW --debug 2>&1)
+test_expect_equal "$output" \
+"(D) add_files_recursive, pass 1: explicitly ignoring /dev/shm/notmuch/tmp.new/mail/.git
+(D) add_files_recursive, pass 1: explicitly ignoring /dev/shm/notmuch/tmp.new/mail/one/two/three/.git
+(D) add_files_recursive, pass 2: explicitly ignoring /dev/shm/notmuch/tmp.new/mail/one/two/three/.git
+(D) add_files_recursive, pass 2: explicitly ignoring /dev/shm/notmuch/tmp.new/mail/one/two/three/ignored_file
+(D) add_files_recursive, pass 2: explicitly ignoring /dev/shm/notmuch/tmp.new/mail/one/two/ignored_file
+(D) add_files_recursive, pass 2: explicitly ignoring /dev/shm/notmuch/tmp.new/mail/one/ignored_file
+(D) add_files_recursive, pass 2: explicitly ignoring /dev/shm/notmuch/tmp.new/mail/.git
+(D) add_files_recursive, pass 2: explicitly ignoring /dev/shm/notmuch/tmp.new/mail/.ignored_hidden_file
+(D) add_files_recursive, pass 2: explicitly ignoring /dev/shm/notmuch/tmp.new/mail/ignored_file
+No new mail."
+
 
 test_done
-- 
1.7.8.1

Re: [PATCH 1/6] cli: update 'new.ignore' config file comment wrt file/directory matching

[David Bremner]

On Sun, 19 Feb 2012 21:47:51 +0100, Pieter Praet <pieter@praet.org> wrote:
> ---
>  notmuch-config.c |    5 ++++-
>  1 files changed, 4 insertions(+), 1 deletions(-)

pushed.

Re: [PATCH 2/6] NEWS: sync 'new.ignore' entry with its comment in notmuch-config.c

[David Bremner]

On Sun, 19 Feb 2012 21:47:52 +0100, Pieter Praet <pieter@praet.org> wrote:
> See previous commit.

Pushed.

d

Re: [PATCH 6/6] test: another test wrt ignoring user-specified files and directories

[David Bremner]

Pieter Praet <pieter@praet.org> writes:

> Demonstrates that *every* file/directory which matches one of the values
> in 'new.ignore' will be ignored, independent of its depth/location in
> the mail store.
> ---

This test fails, apparently because it hard codes the test file paths.

Otherwise the remainder of the series looks fine to me.

d

[PATCH] test: another test wrt ignoring user-specified files and directories

[Ethan Glasser-Camp]

From: Pieter Praet <pieter@praet.org>

Demonstrates that *every* file/directory which matches one of the values
in 'new.ignore' will be ignored, independent of its depth/location in
the mail store.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
---
This is the trivial modification of Pieter's patch that makes it work
no matter where your test directory is.

 test/new |   19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/test/new b/test/new
index cab7c01..cc2af72 100755
--- a/test/new
+++ b/test/new
@@ -183,5 +183,24 @@ touch "${MAIL_DIR}"/.git # change .git's mtime for notmuch new to rescan.
 output=$(NOTMUCH_NEW 2>&1)
 test_expect_equal "$output" "Added 1 new message to the database."
 
+test_begin_subtest "Ignore files and directories specified in new.ignore (multiple occurrences)"
+notmuch config set new.ignore .git ignored_file .ignored_hidden_file
+touch "${MAIL_DIR}"/.git # change .git's mtime for notmuch new to rescan.
+mkdir -p "${MAIL_DIR}"/one/two/three/.git
+notmuch new > /dev/null # ensure that files/folders will be printed in ASCII order.
+touch "${MAIL_DIR}"/{one,one/two,one/two/three}/ignored_file
+output=$(NOTMUCH_NEW --debug 2>&1)
+test_expect_equal "$output" \
+"(D) add_files_recursive, pass 1: explicitly ignoring ${MAIL_DIR}/.git
+(D) add_files_recursive, pass 1: explicitly ignoring ${MAIL_DIR}/one/two/three/.git
+(D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/one/two/three/.git
+(D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/one/two/three/ignored_file
+(D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/one/two/ignored_file
+(D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/one/ignored_file
+(D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/.git
+(D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/.ignored_hidden_file
+(D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/ignored_file
+No new mail."
+
 
 test_done
-- 
1.7.9.5

Re: [PATCH] test: another test wrt ignoring user-specified files and directories

[David Bremner]

Ethan Glasser-Camp <ethan.glasser.camp@gmail.com> writes:
> This is the trivial modification of Pieter's patch that makes it work
> no matter where your test directory is.

pushed patches 3 though 6,

d

And yes, _this_ is the series where I used Ethan's edited patch.