Problem with "Unexpected output" messages

Problem with "Unexpected output" messages

[Andreas Amann]

since commit 44d3c57e (emacs: Display any unexpected output from notmuch
search) I see a number of messages of the form

Error: Unexpected output from notmuch search:
thread:000000000000XXXX 

after notmuch-search in emacs. 

The problem seems to be that some messages contain a "CR LF" sequence in
the "Subject:" line which is either QP or base64 encoded. Apparently the
emacs part of notmuch does not expect such a newline in the subject and
cannot parse the output of "notmuch search". I am happy to forward one
of the offending messages at request. 

This problem occurs with the git master branch as well as with the
crypto branch. Is there any way around this?

Andreas

Re: Problem with "Unexpected output" messages

[Andreas Amann]

On Wed, 06 Apr 2011 20:23:17 +0100, Andreas Amann <a.amann@ucc.ie> wrote:
> 
> since commit 44d3c57e (emacs: Display any unexpected output from notmuch
> search) I see a number of messages of the form
> 
> Error: Unexpected output from notmuch search:
> thread:000000000000XXXX 
> 
> after notmuch-search in emacs. 


FWIW, the patch below solves the problem for me.

Andreas

---------------------
[PATCH] Sanitize "Subject:" and "Author:" fields to not contain control characters in notmuch-search

When a Subject field contained encoded CRLF sequences, these sequences
would appear unfiltered in the output of notmuch search. This confused
the notmuch emacs interface leading to "Unexpected Output"
messages. This is now fixed by replacing all characters with ASCII
code less than 32 with a question mark.
---
 notmuch-search.c |   22 ++++++++++++++++++++--
 1 files changed, 20 insertions(+), 2 deletions(-)

diff --git a/notmuch-search.c b/notmuch-search.c
index 8b90121..fc13e60 100644
--- a/notmuch-search.c
+++ b/notmuch-search.c
@@ -108,6 +108,20 @@ format_item_id_text (unused (const void *ctx),
     printf ("%s%s", item_type, item_id);
 }
 
+static char *
+sanitize_string(const void *ctx, const char *str)
+{
+    char *out, *loop;
+
+    loop = out = talloc_strdup (ctx, str);
+
+    for(;*loop;loop++){
+	if ((unsigned char)(*loop) < 32)
+	    *loop = '?';
+    }
+    return out;
+}
+
 static void
 format_thread_text (const void *ctx,
 		    const char *thread_id,
@@ -117,13 +131,17 @@ format_thread_text (const void *ctx,
 		    const char *authors,
 		    const char *subject)
 {
+    void *ctx_quote = talloc_new (ctx);
+
     printf ("thread:%s %12s [%d/%d] %s; %s",
 	    thread_id,
 	    notmuch_time_relative_date (ctx, date),
 	    matched,
 	    total,
-	    authors,
-	    subject);
+	    sanitize_string(ctx_quote, authors),
+	    sanitize_string(ctx_quote, subject));
+
+    talloc_free (ctx_quote);
 }
 
 static void
-- 
1.7.4.1

Re: Problem with "Unexpected output" messages

[Florian Friesdorf]

[-- Attachment #1: Type: text/plain, Size: 1050 bytes --]


On Fri, 15 Apr 2011 12:28:00 +0100, Andreas Amann <a.amann@ucc.ie> wrote:
> On Wed, 06 Apr 2011 20:23:17 +0100, Andreas Amann <a.amann@ucc.ie> wrote:
> > 
> > since commit 44d3c57e (emacs: Display any unexpected output from notmuch
> > search) I see a number of messages of the form
> > 
> > Error: Unexpected output from notmuch search:
> > thread:000000000000XXXX 
> > 
> > after notmuch-search in emacs. 
> 
> FWIW, the patch below solves the problem for me.

Thx, the patch enables display and handling of a message with which I
had the same problem.

In the message list (emacs UI) the subject now shows as:
foo?

When showing the mail, the buffer name is:
foo^J

and there is an empty line in the header part as the control character
is still present:
Subject: foo

To: my@email
Date: ...

Could/should this also be changed?

-- 
Florian Friesdorf <flo@chaoflow.net>
  GPG FPR: 7A13 5EEE 1421 9FC2 108D  BAAF 38F8 99A3 0C45 F083
Jabber/XMPP: flo@chaoflow.net
IRC: chaoflow on freenode,ircnet,blafasel,OFTC

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: Problem with "Unexpected output" messages

[Pieter Praet]

On Fri, 15 Apr 2011 12:28:00 +0100, Andreas Amann <a.amann@ucc.ie> wrote:
> On Wed, 06 Apr 2011 20:23:17 +0100, Andreas Amann <a.amann@ucc.ie> wrote:
> > 
> > since commit 44d3c57e (emacs: Display any unexpected output from notmuch
> > search) I see a number of messages of the form
> > 
> > Error: Unexpected output from notmuch search:
> > thread:000000000000XXXX 
> > 
> > after notmuch-search in emacs. 
> 
> 
> FWIW, the patch below solves the problem for me.
> 
> Andreas
> 
> ---------------------
> [PATCH] Sanitize "Subject:" and "Author:" fields to not contain control characters in notmuch-search
> 
> When a Subject field contained encoded CRLF sequences, these sequences
> would appear unfiltered in the output of notmuch search. This confused
> the notmuch emacs interface leading to "Unexpected Output"
> messages. This is now fixed by replacing all characters with ASCII
> code less than 32 with a question mark.
> ---
>  notmuch-search.c |   22 ++++++++++++++++++++--
>  1 files changed, 20 insertions(+), 2 deletions(-)
> 
> diff --git a/notmuch-search.c b/notmuch-search.c
> index 8b90121..fc13e60 100644
> --- a/notmuch-search.c
> +++ b/notmuch-search.c
> @@ -108,6 +108,20 @@ format_item_id_text (unused (const void *ctx),
>      printf ("%s%s", item_type, item_id);
>  }
>  
> +static char *
> +sanitize_string(const void *ctx, const char *str)
> +{
> +    char *out, *loop;
> +
> +    loop = out = talloc_strdup (ctx, str);
> +
> +    for(;*loop;loop++){
> +	if ((unsigned char)(*loop) < 32)
> +	    *loop = '?';
> +    }
> +    return out;
> +}
> +
>  static void
>  format_thread_text (const void *ctx,
>  		    const char *thread_id,
> @@ -117,13 +131,17 @@ format_thread_text (const void *ctx,
>  		    const char *authors,
>  		    const char *subject)
>  {
> +    void *ctx_quote = talloc_new (ctx);
> +
>      printf ("thread:%s %12s [%d/%d] %s; %s",
>  	    thread_id,
>  	    notmuch_time_relative_date (ctx, date),
>  	    matched,
>  	    total,
> -	    authors,
> -	    subject);
> +	    sanitize_string(ctx_quote, authors),
> +	    sanitize_string(ctx_quote, subject));
> +
> +    talloc_free (ctx_quote);
>  }
>  
>  static void
> -- 
> 1.7.4.1
> 
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch


Works as advertised, but IMHO (since notmuch solely does indexing,
search and tagging) notmuch should return messages as truthful
(i.e. unaltered) as possible.

Any postprocessing should be the responsability of the MUA.

However, since notmuch already provides output formats, this could be
made part of those, eg. "--format=sanitized_text".


Peace

-Pieter

Re: Problem with "Unexpected output" messages

[Andreas Amann]

On Sun, 17 Apr 2011 17:56:21 +0200, Florian Friesdorf <flo@chaoflow.net> wrote:
> 
> On Fri, 15 Apr 2011 12:28:00 +0100, Andreas Amann <a.amann@ucc.ie> wrote:
> > On Wed, 06 Apr 2011 20:23:17 +0100, Andreas Amann <a.amann@ucc.ie> wrote:
> > > 
> > > since commit 44d3c57e (emacs: Display any unexpected output from notmuch
> > > search) I see a number of messages of the form
> > > 
> > > Error: Unexpected output from notmuch search:
> > > thread:000000000000XXXX 
> > > 
> > > after notmuch-search in emacs. 
> > 
> > FWIW, the patch below solves the problem for me.
> 
> Thx, the patch enables display and handling of a message with which I
> had the same problem.
> 
> In the message list (emacs UI) the subject now shows as:
> foo?
> 
> When showing the mail, the buffer name is:
> foo^J
> 
> and there is an empty line in the header part as the control character
> is still present:
> Subject: foo
> 
> To: my@email
> Date: ...
> 
> Could/should this also be changed?

It could of course. But maybe that should be the responsibility of the
emacs interface?

Re: Problem with "Unexpected output" messages

[Pieter Praet]

[-- Attachment #1: Type: message/rfc822, Size: 2104 bytes --]

From: Pieter Praet <pieter@praet.org>
To: Andreas Amann <a.amann@ucc.ie>
Subject: Re: Problem with "Unexpected output" messages
Date: Mon, 18 Apr 2011 10:52:38 +0200
Message-ID: <87pqokx7op.fsf@A7GMS.i-did-not-set--mail-host-address--so-tickle-me>

On Mon, 18 Apr 2011 09:10:45 +0100, Andreas Amann <a.amann@ucc.ie> wrote:
> On Sun, 17 Apr 2011 19:24:16 +0200, Pieter Praet <pieter@praet.org> wrote:
> > On Fri, 15 Apr 2011 12:28:00 +0100, Andreas Amann <a.amann@ucc.ie> wrote:
> > > On Wed, 06 Apr 2011 20:23:17 +0100, Andreas Amann <a.amann@ucc.ie> wrote:
> > > > 
> > > > since commit 44d3c57e (emacs: Display any unexpected output from notmuch
> > > > search) I see a number of messages of the form
> > > > 
> > > > Error: Unexpected output from notmuch search:
> > > > thread:000000000000XXXX 
> > > > 
> > > > after notmuch-search in emacs. 
> > > 
> > > 
> > > FWIW, the patch below solves the problem for me.
> > > 
> > 
> > 
> > Works as advertised, but IMHO (since notmuch solely does indexing,
> > search and tagging) notmuch should return messages as truthful
> > (i.e. unaltered) as possible.
> > 
> > Any postprocessing should be the responsability of the MUA.
> > 
> > However, since notmuch already provides output formats, this could be
> > made part of those, eg. "--format=sanitized_text".
> 
> In my opinion, "notmuch search" should output precisely one line per
> message per default, and if it outputs two or more lines for one message
> it should be considered a bug.

Agreed. I got carried away after reading Florian's reply [1].

> 
> That's a different story for "notmuch show" where I agree that
> "--format=..." could be useful to get sanitized versions. 
> 
> Andreas
> 


Peace

-Pieter


[1] id:"87d3kkdg7u.fsf@eve.chaoflow.net"

[-- Attachment #2: Type: text/plain, Size: 13 bytes --]



-- 
Pieter