[PATCH v4 0/3] indicate length of omitted body content

[PATCH v4 0/3] indicate length of omitted body content

[Mark Walters]

This is a (trivially) rebased version of
id:1344428872-12374-1-git-send-email-novalazy@gmail.com (There was one
bit of context clash, and a small change to one of the sexp tests to
mirror the same change for the json test)


The patch is Peter's not mine so as a reviewer +1 from me.

Best wishes

Mark


Peter Wang (3):
  test: normalize only message filenames in show json
  show: indicate length, encoding of omitted body content
  test: conform to content length, encoding fields

 devel/schemata   |    9 ++++++++-
 notmuch-show.c   |   14 ++++++++++++++
 test/crypto      |   30 +++++++++++++++++++++---------
 test/json        |    4 +++-
 test/multipart   |   11 ++++++-----
 test/sexp        |    2 +-
 test/test-lib.sh |    2 +-
 7 files changed, 54 insertions(+), 18 deletions(-)

-- 
1.7.9.1

[PATCH v4 1/3] test: normalize only message filenames in show json

[Mark Walters]

From: Peter Wang <novalazy@gmail.com>

notmuch_json_show_sanitize replaced "filename" field values even in part
structures, where the value is predictable.  Make it only normalize the
filename value if it is an absolute path (begins with slash), which is
true of the Maildir filenames that were intended to be normalized away.
---
 test/multipart   |    2 +-
 test/test-lib.sh |    2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/test/multipart b/test/multipart
index 0527f84..344ed81 100755
--- a/test/multipart
+++ b/test/multipart
@@ -630,7 +630,7 @@ cat <<EOF >EXPECTED
  "content": "This is an embedded message, with a multipart/alternative part.\n"}]}]}]},
  {"id": 7,
  "content-type": "text/plain",
- "filename": "YYYYY",
+ "filename": "attachment",
  "content": "This is a text attachment.\n"},
  {"id": 8,
  "content-type": "text/plain",
diff --git a/test/test-lib.sh b/test/test-lib.sh
index fd64736..6ce3b31 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -600,7 +600,7 @@ notmuch_json_show_sanitize ()
 {
     sed \
 	-e 's|"id": "[^"]*",|"id": "XXXXX",|g' \
-	-e 's|"filename": "[^"]*",|"filename": "YYYYY",|g'
+	-e 's|"filename": "/[^"]*",|"filename": "YYYYY",|g'
 }
 
 # End of notmuch helper functions
-- 
1.7.9.1

[PATCH v4 2/3] show: indicate length, encoding of omitted body content

[Mark Walters]

From: Peter Wang <novalazy@gmail.com>

If a leaf part's body content is omitted, return the encoded length and
transfer encoding in --format=json output.  This information may be used
by the consumer, e.g. to decide whether to download a large attachment
over a slow link.

Returning the _encoded_ content length is more efficient than returning
the _decoded_ content length.  Returning the transfer encoding allows
the consumer to estimate the decoded content length.
---
 devel/schemata |    9 ++++++++-
 notmuch-show.c |   14 ++++++++++++++
 2 files changed, 22 insertions(+), 1 deletions(-)

diff --git a/devel/schemata b/devel/schemata
index d1ab983..7d167be 100644
--- a/devel/schemata
+++ b/devel/schemata
@@ -75,7 +75,14 @@ part = {
     # A leaf part's body content is optional, but may be included if
     # it can be correctly encoded as a string.  Consumers should use
     # this in preference to fetching the part content separately.
-    content?:       string
+    content?:       string,
+    # If a leaf part's body content is not included, the length of
+    # the encoded content (in bytes) may be given instead.
+    content-length?: int,
+    # If a leaf part's body content is not included, its transfer encoding
+    # may be given.  Using this and the encoded content length, it is
+    # possible for the consumer to estimate the decoded content length.
+    content-transfer-encoding?: string
 }
 
 # The headers of a message or part (format_headers_sprinter with reply = FALSE)
diff --git a/notmuch-show.c b/notmuch-show.c
index a83fef9..4aeb949 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -692,6 +692,20 @@ format_part_sprinter (const void *ctx, sprinter_t *sp, mime_node_t *node,
 	    sp->map_key (sp, "content");
 	    sp->string_len (sp, (char *) part_content->data, part_content->len);
 	    g_object_unref (stream_memory);
+	} else {
+	    GMimeDataWrapper *wrapper = g_mime_part_get_content_object (GMIME_PART (node->part));
+	    GMimeStream *stream = g_mime_data_wrapper_get_stream (wrapper);
+	    ssize_t length = g_mime_stream_length (stream);
+	    const char *cte = g_mime_object_get_header (meta, "content-transfer-encoding");
+
+	    if (length >= 0) {
+		sp->map_key (sp, "content-length");
+		sp->integer (sp, length);
+	    }
+	    if (cte) {
+		sp->map_key (sp, "content-transfer-encoding");
+		sp->string (sp, cte);
+	    }
 	}
     } else if (GMIME_IS_MULTIPART (node->part)) {
 	sp->map_key (sp, "content");
-- 
1.7.9.1

[PATCH v4 3/3] test: conform to content length, encoding fields

[Mark Walters]

From: Peter Wang <novalazy@gmail.com>

Update tests to expect content-length and content-transfer-encoding
fields in show --format=json output, for leaf parts with omitted body
content.
---
 test/crypto    |   30 +++++++++++++++++++++---------
 test/json      |    4 +++-
 test/multipart |    9 +++++----
 test/sexp      |    2 +-
 4 files changed, 30 insertions(+), 15 deletions(-)

diff --git a/test/crypto b/test/crypto
index 5dd14c4..aa96ec2 100755
--- a/test/crypto
+++ b/test/crypto
@@ -61,7 +61,8 @@ expected='[[[{"id": "XXXXX",
  "content-type": "text/plain",
  "content": "This is a test signed message.\n"},
  {"id": 3,
- "content-type": "application/pgp-signature"}]}]},
+ "content-type": "application/pgp-signature",
+ "content-length": 315}]}]},
  []]]]'
 test_expect_equal_json \
     "$output" \
@@ -95,7 +96,8 @@ expected='[[[{"id": "XXXXX",
  "content-type": "text/plain",
  "content": "This is a test signed message.\n"},
  {"id": 3,
- "content-type": "application/pgp-signature"}]}]},
+ "content-type": "application/pgp-signature",
+ "content-length": 315}]}]},
  []]]]'
 test_expect_equal_json \
     "$output" \
@@ -127,7 +129,8 @@ expected='[[[{"id": "XXXXX",
  "content-type": "text/plain",
  "content": "This is a test signed message.\n"},
  {"id": 3,
- "content-type": "application/pgp-signature"}]}]},
+ "content-type": "application/pgp-signature",
+ "content-length": 315}]}]},
  []]]]'
 test_expect_equal_json \
     "$output" \
@@ -196,7 +199,8 @@ expected='[[[{"id": "XXXXX",
  "sigstatus": [],
  "content-type": "multipart/encrypted",
  "content": [{"id": 2,
- "content-type": "application/pgp-encrypted"},
+ "content-type": "application/pgp-encrypted",
+ "content-length": 11},
  {"id": 3,
  "content-type": "multipart/mixed",
  "content": [{"id": 4,
@@ -204,6 +208,8 @@ expected='[[[{"id": "XXXXX",
  "content": "This is a test encrypted message.\n"},
  {"id": 5,
  "content-type": "application/octet-stream",
+ "content-length": 28,
+ "content-transfer-encoding": "base64",
  "filename": "TESTATTACHMENT"}]}]}]},
  []]]]'
 test_expect_equal_json \
@@ -231,9 +237,11 @@ test_expect_equal_file OUTPUT TESTATTACHMENT
 
 test_begin_subtest "decryption failure with missing key"
 mv "${GNUPGHOME}"{,.bak}
+# The length of the encrypted attachment varies so must be normalized.
 output=$(notmuch show --format=json --decrypt subject:"test encrypted message 001" \
     | notmuch_json_show_sanitize \
-    | sed -e 's|"created": [1234567890]*|"created": 946728000|')
+    | sed -e 's|"created": [1234567890]*|"created": 946728000|' \
+    | sed -e 's|"content-length": 6[1234567890]*|"content-length": 652|')
 expected='[[[{"id": "XXXXX",
  "match": true,
  "excluded": false,
@@ -249,9 +257,11 @@ expected='[[[{"id": "XXXXX",
  "encstatus": [{"status": "bad"}],
  "content-type": "multipart/encrypted",
  "content": [{"id": 2,
- "content-type": "application/pgp-encrypted"},
+ "content-type": "application/pgp-encrypted",
+ "content-length": 11},
  {"id": 3,
- "content-type": "application/octet-stream"}]}]},
+ "content-type": "application/octet-stream",
+ "content-length": 652}]}]},
  []]]]'
 test_expect_equal_json \
     "$output" \
@@ -287,7 +297,8 @@ expected='[[[{"id": "XXXXX",
  "userid": " Notmuch Test Suite <test_suite@notmuchmail.org> (INSECURE!)"}],
  "content-type": "multipart/encrypted",
  "content": [{"id": 2,
- "content-type": "application/pgp-encrypted"},
+ "content-type": "application/pgp-encrypted",
+ "content-length": 11},
  {"id": 3,
  "content-type": "text/plain",
  "content": "This is another test encrypted message.\n"}]}]},
@@ -342,7 +353,8 @@ expected='[[[{"id": "XXXXX",
  "content-type": "text/plain",
  "content": "This is a test signed message.\n"},
  {"id": 3,
- "content-type": "application/pgp-signature"}]}]},
+ "content-type": "application/pgp-signature",
+ "content-length": 315}]}]},
  []]]]'
 test_expect_equal_json \
     "$output" \
diff --git a/test/json b/test/json
index bfafd55..ff7fbe1 100755
--- a/test/json
+++ b/test/json
@@ -45,7 +45,9 @@ emacs_deliver_message \
      (insert \"Message-ID: <$id>\n\")"
 output=$(notmuch show --format=json "id:$id")
 filename=$(notmuch search --output=files "id:$id")
-test_expect_equal_json "$output" "[[[{\"id\": \"$id\", \"match\": true, \"excluded\": false, \"filename\": \"$filename\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\"], \"headers\": {\"Subject\": \"$subject\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"test_suite@notmuchmail.org\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 +0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"multipart/mixed\", \"content\": [{\"id\": 2, \"content-type\": \"text/plain\", \"content\": \"This is a test message with inline attachment with a filename\"}, {\"id\": 3, \"content-type\": \"application/octet-stream\", \"filename\": \"README\"}]}]}, []]]]"
+# Get length of README after base64-encoding, minus additional newline.
+attachment_length=$(( $(base64 $TEST_DIRECTORY/README | wc -c) - 1 ))
+test_expect_equal_json "$output" "[[[{\"id\": \"$id\", \"match\": true, \"excluded\": false, \"filename\": \"$filename\", \"timestamp\": 946728000, \"date_relative\": \"2000-01-01\", \"tags\": [\"inbox\"], \"headers\": {\"Subject\": \"$subject\", \"From\": \"Notmuch Test Suite <test_suite@notmuchmail.org>\", \"To\": \"test_suite@notmuchmail.org\", \"Date\": \"Sat, 01 Jan 2000 12:00:00 +0000\"}, \"body\": [{\"id\": 1, \"content-type\": \"multipart/mixed\", \"content\": [{\"id\": 2, \"content-type\": \"text/plain\", \"content\": \"This is a test message with inline attachment with a filename\"}, {\"id\": 3, \"content-type\": \"application/octet-stream\", \"content-length\": $attachment_length, \"content-transfer-encoding\": \"base64\", \"filename\": \"README\"}]}]}, []]]]"
 
 test_begin_subtest "Search message: json, utf-8"
 add_message "[subject]=\"json-search-utf8-body-sübjéct\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"jsön-search-méssage\""
diff --git a/test/multipart b/test/multipart
index 344ed81..497f3ec 100755
--- a/test/multipart
+++ b/test/multipart
@@ -330,7 +330,7 @@ cat <<EOF >EXPECTED
 {"id": 6, "content-type": "text/plain", "content": "This is an embedded message, with a multipart/alternative part.\n"}]}]}]}, 
 {"id": 7, "content-type": "text/plain", "filename": "attachment", "content": "This is a text attachment.\n"}, 
 {"id": 8, "content-type": "text/plain", "content": "And this message is signed.\n\n-Carl\n"}]}, 
-{"id": 9, "content-type": "application/pgp-signature"}]}]}
+{"id": 9, "content-type": "application/pgp-signature", "content-length": 197}]}]}
 EOF
 test_expect_equal_json "$(cat OUTPUT)" "$(cat EXPECTED)"
 
@@ -345,7 +345,7 @@ cat <<EOF >EXPECTED
 {"id": 6, "content-type": "text/plain", "content": "This is an embedded message, with a multipart/alternative part.\n"}]}]}]}, 
 {"id": 7, "content-type": "text/plain", "filename": "attachment", "content": "This is a text attachment.\n"}, 
 {"id": 8, "content-type": "text/plain", "content": "And this message is signed.\n\n-Carl\n"}]}, 
-{"id": 9, "content-type": "application/pgp-signature"}]}
+{"id": 9, "content-type": "application/pgp-signature", "content-length": 197}]}
 EOF
 test_expect_equal_json "$(cat OUTPUT)" "$(cat EXPECTED)"
 
@@ -412,7 +412,7 @@ test_expect_equal_json "$(cat OUTPUT)" "$(cat EXPECTED)"
 test_begin_subtest "--format=json --part=9, pgp signature (unverified)"
 notmuch show --format=json --part=9 'id:87liy5ap00.fsf@yoom.home.cworth.org' >OUTPUT
 cat <<EOF >EXPECTED
-{"id": 9, "content-type": "application/pgp-signature"}
+{"id": 9, "content-type": "application/pgp-signature", "content-length": 197}
 EOF
 test_expect_equal_json "$(cat OUTPUT)" "$(cat EXPECTED)"
 
@@ -636,7 +636,8 @@ cat <<EOF >EXPECTED
  "content-type": "text/plain",
  "content": "And this message is signed.\n\n-Carl\n"}]},
  {"id": 9,
- "content-type": "application/pgp-signature"}]}]}}
+ "content-type": "application/pgp-signature",
+ "content-length": 197}]}]}}
 EOF
 test_expect_equal_json "$(cat OUTPUT)" "$(cat EXPECTED)"
 
diff --git a/test/sexp b/test/sexp
index b804942..dc52e79 100755
--- a/test/sexp
+++ b/test/sexp
@@ -37,7 +37,7 @@ emacs_deliver_message \
      (insert \"Message-ID: <$id>\n\")"
 output=$(notmuch show --format=sexp "id:$id")
 filename=$(notmuch search --output=files "id:$id")
-test_expect_equal "$output" "((((:id \"$id\" :match t :excluded nil :filename \"$filename\" :timestamp 946728000 :date_relative \"2000-01-01\" :tags (\"inbox\") :headers (:Subject \"sexp-show-inline-attachment-filename\" :From \"Notmuch Test Suite <test_suite@notmuchmail.org>\" :To \"test_suite@notmuchmail.org\" :Date \"Sat, 01 Jan 2000 12:00:00 +0000\") :body ((:id 1 :content-type \"multipart/mixed\" :content ((:id 2 :content-type \"text/plain\" :content \"This is a test message with inline attachment with a filename\") (:id 3 :content-type \"application/octet-stream\" :filename \"README\"))))) ())))"
+test_expect_equal "$output" "((((:id \"$id\" :match t :excluded nil :filename \"$filename\" :timestamp 946728000 :date_relative \"2000-01-01\" :tags (\"inbox\") :headers (:Subject \"sexp-show-inline-attachment-filename\" :From \"Notmuch Test Suite <test_suite@notmuchmail.org>\" :To \"test_suite@notmuchmail.org\" :Date \"Sat, 01 Jan 2000 12:00:00 +0000\") :body ((:id 1 :content-type \"multipart/mixed\" :content ((:id 2 :content-type \"text/plain\" :content \"This is a test message with inline attachment with a filename\") (:id 3 :content-type \"application/octet-stream\" :filename \"README\" :content-length 12555 :content-transfer-encoding \"base64\"))))) ())))"
 
 test_begin_subtest "Search message: sexp, utf-8"
 add_message "[subject]=\"sexp-search-utf8-body-sübjéct\"" "[date]=\"Sat, 01 Jan 2000 12:00:00 -0000\"" "[body]=\"jsön-search-méssage\""
-- 
1.7.9.1

Re: [PATCH v4 0/3] indicate length of omitted body content

[Mark Walters]

On Sun, 09 Dec 2012, Mark Walters <markwalters1009@gmail.com> wrote:
> This is a (trivially) rebased version of
> id:1344428872-12374-1-git-send-email-novalazy@gmail.com (There was one
> bit of context clash, and a small change to one of the sexp tests to
> mirror the same change for the json test)
>
>
> The patch is Peter's not mine so as a reviewer +1 from me.
>

Hi (sorry Peter I had assumed git send-email would have cc'd you on my
rebase earlier today)

There was some discussion about this patch series on irc and the general
view was favourable but that, for consistency, all omitted parts should
received a content-length (including text/html). This is a little messy
to code into the if else stuff: Jani suggested putting
content-length/content-encoding into a small function.

Anyway I had hope it was just a trivial rebase and could go in, but the
above will take a bit longer to do neatly so I will leave it for now.

I think it could go in relatively easily if resubmitted with this
change but I will mark this version `moreinfo'.


Best wishes

Mark

> Best wishes
>
> Mark
>
>
> Peter Wang (3):
>   test: normalize only message filenames in show json
>   show: indicate length, encoding of omitted body content
>   test: conform to content length, encoding fields
>
>  devel/schemata   |    9 ++++++++-
>  notmuch-show.c   |   14 ++++++++++++++
>  test/crypto      |   30 +++++++++++++++++++++---------
>  test/json        |    4 +++-
>  test/multipart   |   11 ++++++-----
>  test/sexp        |    2 +-
>  test/test-lib.sh |    2 +-
>  7 files changed, 54 insertions(+), 18 deletions(-)
>
> -- 
> 1.7.9.1

Re: [PATCH v4 0/3] indicate length of omitted body content

[Peter Wang]

On Sun, 09 Dec 2012 19:21:18 +0000, Mark Walters <markwalters1009@gmail.com> wrote:
> 
> Hi (sorry Peter I had assumed git send-email would have cc'd you on my
> rebase earlier today)
> 
> There was some discussion about this patch series on irc and the general
> view was favourable but that, for consistency, all omitted parts should
> received a content-length (including text/html). This is a little messy
> to code into the if else stuff: Jani suggested putting
> content-length/content-encoding into a small function.
> 
> Anyway I had hope it was just a trivial rebase and could go in, but the
> above will take a bit longer to do neatly so I will leave it for now.

Ok. To save any potential duplicated work: I've updated the patch here,
save for test/sexp.  Waiting for that to be pushed before updating it.

Peter

Re: [PATCH v4 0/3] indicate length of omitted body content

[Mark Walters]

On Thu, 13 Dec 2012, Peter Wang <novalazy@gmail.com> wrote:
> On Sun, 09 Dec 2012 19:21:18 +0000, Mark Walters <markwalters1009@gmail.com> wrote:
>> 
>> Hi (sorry Peter I had assumed git send-email would have cc'd you on my
>> rebase earlier today)
>> 
>> There was some discussion about this patch series on irc and the general
>> view was favourable but that, for consistency, all omitted parts should
>> received a content-length (including text/html). This is a little messy
>> to code into the if else stuff: Jani suggested putting
>> content-length/content-encoding into a small function.
>> 
>> Anyway I had hope it was just a trivial rebase and could go in, but the
>> above will take a bit longer to do neatly so I will leave it for now.
>
> Ok. To save any potential duplicated work: I've updated the patch here,
> save for test/sexp.  Waiting for that to be pushed before updating it.

Sorry I am confused: isn't test/sexp already in master?

Best wishes 

Mark

Re: [PATCH v4 0/3] indicate length of omitted body content

[Peter Wang]

On Thu, 13 Dec 2012 12:52:50 +0000, Mark Walters <markwalters1009@gmail.com> wrote:
> 
> Sorry I am confused: isn't test/sexp already in master?

You're right, somehow I missed it.

Peter