[PATCH] Support for printing file paths in new command

[PATCH] Support for printing file paths in new command

[Adrian Perez]

For very large mail boxes, it is desirable to know which files are being
processed e.g. when a crash occurs to know which one was the cause. Also,
it may be interesting to have a better idea of how the operation is
progressing when processing mailboxes with big messages.

This patch adds support for printing messages as they are processed by
"notmuch new":

* The "new" command now supports a "--verbose" flag.

* When running in verbose mode, the file path of the message about to be
  processed is printed in the following format:

    current/total: /path/to/message/file

  Where "current" is the number of messages processed so far and "total" is
  the total count of files to be processed.

  The status line is erased using an ANSI sequence "\033[K" (erase current
  line from the cursor to the end of line) each time it is refreshed. This
  should not pose a problem because nearly every terminal supports it.

* The signal handler for SIGALRM and the timer are not enabled when running
  in verbose mode, because we are already printing progress with each file,
  periodical reports are not neccessary.
---
 notmuch-client.h |    1 +
 notmuch-new.c    |   73 +++++++++++++++++++++++++++++++++++------------------
 notmuch.c        |   11 ++++++-
 3 files changed, 58 insertions(+), 27 deletions(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index b65aa77..6908844 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -74,6 +74,7 @@ typedef void (*add_files_callback_t) (notmuch_message_t *message);
 typedef struct {
     int ignore_read_only_directories;
     int saw_read_only_directory;
+    int verbose;
 
     int total_files;
     int processed_files;
diff --git a/notmuch-new.c b/notmuch-new.c
index 1b05584..6b4b83c 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -177,6 +177,14 @@ add_files_recursive (notmuch_database_t *notmuch,
 	    if (path_dbtime == 0 || st->st_mtime > path_dbtime) {
 		state->processed_files++;
 
+		if (state->verbose) {
+		    printf ("\r\033[K%i/%i: %s\r",
+			    state->processed_files,
+			    state->total_files,
+			    next);
+		    fflush (stdout);
+		}
+
 		status = notmuch_database_add_message (notmuch, next, &message);
 		switch (status) {
 		    /* success */
@@ -270,32 +278,36 @@ add_files (notmuch_database_t *notmuch,
 	return NOTMUCH_STATUS_FILE_ERROR;
     }
 
-    /* Setup our handler for SIGALRM */
-    memset (&action, 0, sizeof (struct sigaction));
-    action.sa_handler = handle_sigalrm;
-    sigemptyset (&action.sa_mask);
-    action.sa_flags = SA_RESTART;
-    sigaction (SIGALRM, &action, NULL);
-
-    /* Then start a timer to send SIGALRM once per second. */
-    timerval.it_interval.tv_sec = 1;
-    timerval.it_interval.tv_usec = 0;
-    timerval.it_value.tv_sec = 1;
-    timerval.it_value.tv_usec = 0;
-    setitimer (ITIMER_REAL, &timerval, NULL);
+    if (!state->verbose) {
+	/* Setup our handler for SIGALRM */
+	memset (&action, 0, sizeof (struct sigaction));
+	action.sa_handler = handle_sigalrm;
+	sigemptyset (&action.sa_mask);
+	action.sa_flags = SA_RESTART;
+	sigaction (SIGALRM, &action, NULL);
+
+	/* Then start a timer to send SIGALRM once per second. */
+	timerval.it_interval.tv_sec = 1;
+	timerval.it_interval.tv_usec = 0;
+	timerval.it_value.tv_sec = 1;
+	timerval.it_value.tv_usec = 0;
+	setitimer (ITIMER_REAL, &timerval, NULL);
+    }
 
     status = add_files_recursive (notmuch, path, &st, state);
 
-    /* Now stop the timer. */
-    timerval.it_interval.tv_sec = 0;
-    timerval.it_interval.tv_usec = 0;
-    timerval.it_value.tv_sec = 0;
-    timerval.it_value.tv_usec = 0;
-    setitimer (ITIMER_REAL, &timerval, NULL);
-
-    /* And disable the signal handler. */
-    action.sa_handler = SIG_IGN;
-    sigaction (SIGALRM, &action, NULL);
+    if (!state->verbose) {
+	/* Now stop the timer. */
+	timerval.it_interval.tv_sec = 0;
+	timerval.it_interval.tv_usec = 0;
+	timerval.it_value.tv_sec = 0;
+	timerval.it_value.tv_usec = 0;
+	setitimer (ITIMER_REAL, &timerval, NULL);
+
+	/* And disable the signal handler. */
+	action.sa_handler = SIG_IGN;
+	sigaction (SIGALRM, &action, NULL);
+    }
 
     return status;
 }
@@ -370,8 +382,7 @@ count_files (const char *path, int *count)
 }
 
 int
-notmuch_new_command (void *ctx,
-		     unused (int argc), unused (char *argv[]))
+notmuch_new_command (void *ctx, int argc, char *argv[])
 {
     notmuch_config_t *config;
     notmuch_database_t *notmuch;
@@ -383,6 +394,18 @@ notmuch_new_command (void *ctx,
     const char *db_path;
     char *dot_notmuch_path;
     struct sigaction action;
+    int i;
+
+    add_files_state.verbose = 0;
+
+    for (i = 0; i < argc && argv[i][0] == '-'; i++) {
+	if (STRNCMP_LITERAL (argv[i], "--verbose") == 0) {
+	    add_files_state.verbose = 1;
+	} else {
+	    fprintf (stderr, "Unrecognized option: %s\n", argv[i]);
+	    return 1;
+	}
+    }
 
     /* Setup our handler for SIGINT */
     memset (&action, 0, sizeof (struct sigaction));
diff --git a/notmuch.c b/notmuch.c
index 5cc8e4c..8cb1310 100644
--- a/notmuch.c
+++ b/notmuch.c
@@ -107,8 +107,8 @@ command_t commands[] = {
       "\t\tInvoking notmuch with no command argument will run setup if\n"
       "\t\tthe setup command has not previously been completed." },
     { "new", notmuch_new_command,
-      NULL,
-      "Find and import new messages to the notmuch database.",
+      "[--verbose]",
+      "\t\tFind and import new messages to the notmuch database.",
       "\t\tScans all sub-directories of the mail directory, performing\n"
       "\t\tfull-text indexing on new messages that are found. Each new\n"
       "\t\tmessage will be tagged as both \"inbox\" and \"unread\".\n"
@@ -122,6 +122,13 @@ command_t commands[] = {
       "\t\tis delivered and you wish to incorporate it into the database.\n"
       "\t\tThese subsequent runs will be much quicker than the initial run.\n"
       "\n"
+      "\t\tSupported options for new include:\n"
+      "\n"
+      "\t\t--verbose\n"
+      "\n"
+      "\t\t\tVerbose operation. Shows paths of message files as\n"
+      "\t\t\tthey are being indexed.\n"
+      "\n"
       "\t\tNote: \"notmuch new\" runs (other than the first run) will\n"
       "\t\tskip any read-only directories, so you can use that to mark\n"
       "\t\tdirectories that will not receive any new mail (and make\n"
-- 
1.6.5.2

[PATCH] Support for printing file paths in new command

[Adrian Perez]

For very large mail boxes, it is desirable to know which files are being
processed e.g. when a crash occurs to know which one was the cause. Also,
it may be interesting to have a better idea of how the operation is
progressing when processing mailboxes with big messages.

This patch adds support for printing messages as they are processed by
"notmuch new":

* The "new" command now supports a "--verbose" flag.

* When running in verbose mode, the file path of the message about to be
  processed is printed in the following format:

    current/total: /path/to/message/file

  Where "current" is the number of messages processed so far and "total" is
  the total count of files to be processed.

* The status line is erased using an ANSI sequence "\033[K" (erase current
  line from the cursor to the end of line) each time it is refreshed. This
  should not pose a problem because nearly every terminal supports it.

* When output is not being sent to a terminal, then each file processed is
  printed in a new line and ANSI escapes are not used at all.

* The signal handler for SIGALRM and the timer are not enabled when running
  in verbose mode, because we are already printing progress with each file,
  periodical reports are not neccessary.
---
 notmuch-client.h |    2 ++
 notmuch-new.c    |   32 +++++++++++++++++++++++++++++---
 notmuch.c        |   11 +++++++++--
 3 files changed, 40 insertions(+), 5 deletions(-)

diff --git a/notmuch-client.h b/notmuch-client.h
index ea77686..4fe182e 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -74,6 +74,8 @@ typedef void (*add_files_callback_t) (notmuch_message_t *message);
 typedef struct {
     int ignore_read_only_directories;
     int saw_read_only_directory;
+    int output_is_a_tty;
+    int verbose;
 
     int total_files;
     int processed_files;
diff --git a/notmuch-new.c b/notmuch-new.c
index 0dd2784..b5c6cc6 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -179,6 +179,19 @@ add_files_recursive (notmuch_database_t *notmuch,
 	    if (path_dbtime == 0 || st->st_mtime > path_dbtime) {
 		state->processed_files++;
 
+		if (state->verbose) {
+		    if (state->output_is_a_tty)
+			printf("\r\033[K");
+
+		    printf ("%i/%i: %s",
+			    state->processed_files,
+			    state->total_files,
+			    next);
+
+		    putchar((state->output_is_a_tty) ? '\r' : '\n');
+		    fflush (stdout);
+		}
+
 		status = notmuch_database_add_message (notmuch, next, &message);
 		switch (status) {
 		    /* success */
@@ -275,7 +288,7 @@ add_files (notmuch_database_t *notmuch,
     }
 
     /* Setup our handler for SIGALRM */
-    if (isatty (fileno (stdout)) && ! debugger_is_active ()) {
+    if (state->output_is_a_tty && ! debugger_is_active () && ! state->verbose) {
 	memset (&action, 0, sizeof (struct sigaction));
 	action.sa_handler = handle_sigalrm;
 	sigemptyset (&action.sa_mask);
@@ -296,6 +309,7 @@ add_files (notmuch_database_t *notmuch,
 
     /* Now stop the timer. */
     if (timer_is_active) {
+	/* Now stop the timer. */
 	timerval.it_interval.tv_sec = 0;
 	timerval.it_interval.tv_usec = 0;
 	timerval.it_value.tv_sec = 0;
@@ -380,8 +394,7 @@ count_files (const char *path, int *count)
 }
 
 int
-notmuch_new_command (void *ctx,
-		     unused (int argc), unused (char *argv[]))
+notmuch_new_command (void *ctx, int argc, char *argv[])
 {
     notmuch_config_t *config;
     notmuch_database_t *notmuch;
@@ -393,6 +406,19 @@ notmuch_new_command (void *ctx,
     const char *db_path;
     char *dot_notmuch_path;
     struct sigaction action;
+    int i;
+
+    add_files_state.verbose = 0;
+    add_files_state.output_is_a_tty = isatty (fileno (stdout));
+
+    for (i = 0; i < argc && argv[i][0] == '-'; i++) {
+	if (STRNCMP_LITERAL (argv[i], "--verbose") == 0) {
+	    add_files_state.verbose = 1;
+	} else {
+	    fprintf (stderr, "Unrecognized option: %s\n", argv[i]);
+	    return 1;
+	}
+    }
 
     /* Setup our handler for SIGINT */
     memset (&action, 0, sizeof (struct sigaction));
diff --git a/notmuch.c b/notmuch.c
index 5cc8e4c..8cb1310 100644
--- a/notmuch.c
+++ b/notmuch.c
@@ -107,8 +107,8 @@ command_t commands[] = {
       "\t\tInvoking notmuch with no command argument will run setup if\n"
       "\t\tthe setup command has not previously been completed." },
     { "new", notmuch_new_command,
-      NULL,
-      "Find and import new messages to the notmuch database.",
+      "[--verbose]",
+      "\t\tFind and import new messages to the notmuch database.",
       "\t\tScans all sub-directories of the mail directory, performing\n"
       "\t\tfull-text indexing on new messages that are found. Each new\n"
       "\t\tmessage will be tagged as both \"inbox\" and \"unread\".\n"
@@ -122,6 +122,13 @@ command_t commands[] = {
       "\t\tis delivered and you wish to incorporate it into the database.\n"
       "\t\tThese subsequent runs will be much quicker than the initial run.\n"
       "\n"
+      "\t\tSupported options for new include:\n"
+      "\n"
+      "\t\t--verbose\n"
+      "\n"
+      "\t\t\tVerbose operation. Shows paths of message files as\n"
+      "\t\t\tthey are being indexed.\n"
+      "\n"
       "\t\tNote: \"notmuch new\" runs (other than the first run) will\n"
       "\t\tskip any read-only directories, so you can use that to mark\n"
       "\t\tdirectories that will not receive any new mail (and make\n"
-- 
1.6.5.2

Re: [PATCH] Support for printing file paths in new command

[Carl Worth]

On Sat, 21 Nov 2009 01:17:18 +0100, Adrian Perez <aperez@igalia.com> wrote:
> For very large mail boxes, it is desirable to know which files are being
> processed e.g. when a crash occurs to know which one was the cause. Also,
> it may be interesting to have a better idea of how the operation is
> progressing when processing mailboxes with big messages.
> 
> This patch adds support for printing messages as they are processed by
> "notmuch new":

Hey, that's pretty nice.

I've pushed this now, (and I'm almost tempted to make it the
default...).

-Carl

Re: [PATCH] Support for printing file paths in new command

[Adrian Perez de Castro]

[-- Attachment #1: Type: text/plain, Size: 1326 bytes --]

On Mon, 23 Nov 2009 01:24:09 +0100, Carl wrote:

> On Sat, 21 Nov 2009 01:17:18 +0100, Adrian Perez <aperez@igalia.com> wrote:
> > For very large mail boxes, it is desirable to know which files are being
> > processed e.g. when a crash occurs to know which one was the cause. Also,
> > it may be interesting to have a better idea of how the operation is
> > progressing when processing mailboxes with big messages.
> > 
> > This patch adds support for printing messages as they are processed by
> > "notmuch new":
> 
> Hey, that's pretty nice.

Thanks!

As a small improvement, I will send another patch in a while, which will
avoid sending ANSI escapes when output is not a terminal, and print one
file path per line in that case.

> I've pushed this now, (and I'm almost tempted to make it the
> default...).

IMHO it would be nice to have the messages-per-second information as
well in verbose mode before making it the default. I think I could cook
up a patch for that, but printing messages-per-second would mean calling
gettimeofday() for each processed file. Do you think that will be
acceptable from a performance POV? (Of course using silent mode would
still only call it once per second).

Regards,

-- 
Adrian Perez de Castro <aperez@igalia.com>
Igalia - Free Software Engineering

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [PATCH] Support for printing file paths in new command

[Carl Worth]

On Mon, 23 Nov 2009 01:53:39 +0100, Adrian Perez de Castro <aperez@igalia.com> wrote:
> IMHO it would be nice to have the messages-per-second information as
> well in verbose mode before making it the default. I think I could cook
> up a patch for that, but printing messages-per-second would mean calling
> gettimeofday() for each processed file. Do you think that will be
> acceptable from a performance POV? (Of course using silent mode would
> still only call it once per second).

Or how about using the existing timer to cause a rate to be printed
between the message output once per second?

-Carl