From: "Rollins, Jameson" <jrollins@caltech.edu>
To: Notmuch Mail <notmuch@notmuchmail.org>
Subject: [PATCH v3] test: new test framework to compare json parts
Date: Mon, 27 May 2019 18:35:10 +0000 [thread overview]
Message-ID: <20190527183508.9843-1-jrollins@caltech.edu> (raw)
In-Reply-To: <20190526221610.2833-4-dkg@fifthhorseman.net>
From: Jameson Graef Rollins <jrollins@finestructure.net>
This makes it easier to write fairly compact, readable tests of json
output, without needing to sanitize away parts that we don't care
about.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
---
test/json_check_nodes.py | 114 +++++++++++++++++++++++++++++++++++++++
test/test-lib.sh | 24 +++++++++
2 files changed, 138 insertions(+)
create mode 100755 test/json_check_nodes.py
diff --git a/test/json_check_nodes.py b/test/json_check_nodes.py
new file mode 100755
index 00000000..17403c57
--- /dev/null
+++ b/test/json_check_nodes.py
@@ -0,0 +1,114 @@
+#!/usr/bin/env python
+import re
+import sys
+import json
+
+
+EXPR_RE = re.compile('(?P<label>[a-zA-Z0-9_-]+):(?P<address>[^=!]+)(?:(?P<type>[=!])(?P<val>.*))?', re.DOTALL|re.MULTILINE)
+
+
+if len(sys.argv) < 2:
+ sys.exit('usage: '+ sys.argv[0] + """ EXPR [EXPR]
+
+Takes json data on stdin and evaluates test expressions specified in
+arguments. Each test is evaluated, and output is printed only if the
+test fails. If any test fails the return value of execution will be
+non-zero.
+
+EXPR can be one of following types:
+
+Value test: test that object in json data found at address is equal to
+specified value:
+
+ label:address=value
+
+Existence test: test that dict or list in json data found at address
+does *not* contain the specified key:
+
+ label:address!key
+
+Extract: extract object from json data found at address and print
+
+ label:address
+
+Results are printed to stdout prefixed by expression label. In all
+cases the test will fail if object does not exist in data.
+
+Example:
+
+0 $ echo '["a", "b", {"c": 1}]' | python3 json_check_nodes.py 'second_d:[1]="d"' 'no_c:[2]!"c"'
+second_d: value not equal: data[1] = 'b' != 'd'
+no_c: dict contains key: data[2]["c"] = 1
+1 $
+
+""")
+
+
+# parse expressions from arguments
+exprs = []
+for expr in sys.argv[1:]:
+ m = re.match(EXPR_RE, expr)
+ if not m:
+ sys.exit("Invalid expression: {}".format(expr))
+ exprs.append(m)
+
+data = json.load(sys.stdin)
+
+fail = False
+
+for expr in exprs:
+ # print(expr.groups(),fail)
+
+ e = 'data{}'.format(expr.group('address'))
+ try:
+ val = eval(e)
+ except SyntaxError:
+ fail = True
+ print("{}: syntax error on evaluation of object: {}".format(
+ expr.group('label'), e))
+ continue
+ except:
+ fail = True
+ print("{}: object not found: data{}".format(
+ expr.group('label'), expr.group('address')))
+ continue
+
+ if expr.group('type') == '=':
+ try:
+ obj_val = json.loads(expr.group('val'))
+ except:
+ fail = True
+ print("{}: error evaluating value: {}".format(
+ expr.group('label'), expr.group('address')))
+ continue
+ if val != obj_val:
+ fail = True
+ print("{}: value not equal: data{} = {} != {}".format(
+ expr.group('label'), expr.group('address'), repr(val), repr(obj_val)))
+
+ elif expr.group('type') == '!':
+ if not isinstance(val, (dict, list)):
+ fail = True
+ print("{}: not a dict or a list: data{}".format(
+ expr.group('label'), expr.group('address')))
+ continue
+ try:
+ idx = json.loads(expr.group('val'))
+ if idx in val:
+ fail = True
+ print("{}: {} contains key: {}[{}] = {}".format(
+ expr.group('label'), type(val).__name__, e, expr.group('val'), val[idx]))
+ except SyntaxError:
+ fail = True
+ print("{}: syntax error on evaluation of value: {}".format(
+ expr.group('label'), expr.group('val')))
+ continue
+
+
+ elif expr.group('type') is None:
+ print("{}: {}".format(expr.group('label'), val))
+
+
+if fail:
+ sys.exit(1)
+sys.exit(0)
diff --git a/test/test-lib.sh b/test/test-lib.sh
index ff18fae6..616cb674 100644
--- a/test/test-lib.sh
+++ b/test/test-lib.sh
@@ -507,6 +507,30 @@ test_sort_json () {
"import sys, json; json.dump(sorted(json.load(sys.stdin)),sys.stdout)"
}
+# test for json objects:
+# read the source of test/json_check_nodes.py (or the output when
+# invoking it without arguments) for an explanation of the syntax.
+test_json_nodes () {
+ exec 1>&6 2>&7 # Restore stdout and stderr
+ if [ -z "$inside_subtest" ]; then
+ error "bug in the test script: test_json_eval without test_begin_subtest"
+ fi
+ inside_subtest=
+ test "$#" > 0 ||
+ error "bug in the test script: test_json_nodes needs at least 1 parameter"
+
+ if ! test_skip "$test_subtest_name"
+ then
+ output=$(PYTHONIOENCODING=utf-8 $NOTMUCH_PYTHON "$TEST_DIRECTORY"/json_check_nodes.py "$@")
+ if [ "$?" = 0 ]
+ then
+ test_ok_
+ else
+ test_failure_ "$output"
+ fi
+ fi
+}
+
test_emacs_expect_t () {
test "$#" = 1 ||
error "bug in the test script: not 1 parameter to test_emacs_expect_t"
--
2.20.1
next prev parent reply other threads:[~2019-05-27 18:35 UTC|newest]
Thread overview: 47+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-05-26 22:15 Protected Headers (2nd major revision, more testing!) Daniel Kahn Gillmor
2019-05-26 22:15 ` [PATCH v2 01/17] cli/show: emit headers after emitting body Daniel Kahn Gillmor
2019-05-26 22:15 ` [PATCH v2 02/17] util/crypto: add information about the payload part Daniel Kahn Gillmor
2019-05-26 22:15 ` [PATCH v2 03/17] test: new test framework to compare json parts Daniel Kahn Gillmor
2019-05-27 9:56 ` David Bremner
2019-05-27 17:31 ` Rollins, Jameson
2019-05-27 20:34 ` [PATCH v3 " Daniel Kahn Gillmor
2019-05-27 21:30 ` Daniel Kahn Gillmor
2019-05-28 0:09 ` Rollins, Jameson
2019-05-27 18:35 ` Rollins, Jameson [this message]
2019-05-26 22:15 ` [PATCH v2 04/17] cli/show: add tests for viewing protected headers Daniel Kahn Gillmor
2019-05-26 22:15 ` [PATCH v2 05/17] cli/show: emit payload subject instead of outside subject Daniel Kahn Gillmor
2019-05-26 22:15 ` [PATCH v2 06/17] cli/show: add information about which headers were protected Daniel Kahn Gillmor
2019-05-27 10:12 ` David Bremner
2019-05-27 17:34 ` Rollins, Jameson
2019-05-27 17:59 ` David Bremner
2019-05-27 20:40 ` [PATCH v3 " Daniel Kahn Gillmor
2019-05-27 22:18 ` Daniel Kahn Gillmor
2019-05-27 20:43 ` [PATCH v2 " Daniel Kahn Gillmor
2019-05-27 22:14 ` [PATCH v4 " Daniel Kahn Gillmor
2019-05-28 11:10 ` David Bremner
2019-05-28 22:39 ` Daniel Kahn Gillmor
2019-05-26 22:16 ` [PATCH v2 07/17] test: add test for missing external subject Daniel Kahn Gillmor
2019-05-26 22:16 ` [PATCH v2 08/17] test: show cryptographic envelope information for signed mails Daniel Kahn Gillmor
2019-05-26 22:16 ` [PATCH v2 09/17] cli/reply: ensure encrypted Subject: line does not leak in the clear Daniel Kahn Gillmor
2019-05-26 22:16 ` [PATCH v2 10/17] indexing: record protected subject when indexing cleartext Daniel Kahn Gillmor
2019-05-27 10:24 ` David Bremner
2019-05-27 21:17 ` [PATCH v3 " Daniel Kahn Gillmor
2019-05-27 22:35 ` Daniel Kahn Gillmor
2019-05-27 21:25 ` _notmuch_database_log vs _notmuch_database_log_append [was: Re: [PATCH v2 10/17] indexing: record protected subject when indexing cleartext] Daniel Kahn Gillmor
2019-05-27 22:40 ` [PATCH v4 10/17] indexing: record protected subject when indexing cleartext Daniel Kahn Gillmor
2019-05-26 22:16 ` [PATCH v2 11/17] test: protected headers should work when both encrypted and signed Daniel Kahn Gillmor
2019-05-26 22:16 ` [PATCH v2 12/17] test: after reindexing, only legitimate protected subjects are searchable Daniel Kahn Gillmor
2019-05-26 22:16 ` [PATCH v2 13/17] test: try indexing nested messages and protected headers Daniel Kahn Gillmor
2019-05-26 22:16 ` [PATCH v2 14/17] test: ensure that protected headers appear in notmuch-emacs search as expected Daniel Kahn Gillmor
2019-05-27 20:21 ` Rollins, Jameson
2019-05-27 21:58 ` [PATCH v3 " Daniel Kahn Gillmor
2019-05-27 22:02 ` stitching threads (v3 14/17) Daniel Kahn Gillmor
2021-12-23 11:57 ` David Bremner
2019-05-26 22:16 ` [PATCH v2 15/17] test: emacs/show: ensure that protected headers appear as expected Daniel Kahn Gillmor
2019-05-26 22:16 ` [PATCH v2 16/17] test: reply (in cli and emacs) should protect indexed sensitive headers Daniel Kahn Gillmor
2019-05-26 22:16 ` [PATCH v2 17/17] cli/reply: pull proposed subject line from the message, not the index Daniel Kahn Gillmor
2019-05-27 20:22 ` Protected Headers (2nd major revision, more testing!) Rollins, Jameson
2019-05-27 22:49 ` Daniel Kahn Gillmor
2019-05-29 11:44 ` David Bremner
2019-05-29 17:31 ` Daniel Kahn Gillmor
[not found] <87d0k3643o.fsf@caltech.edu.net>
2019-05-27 21:35 ` [PATCH v3 14/17] test: ensure that protected headers appear in notmuch-emacs search as expected Daniel Kahn Gillmor
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://notmuchmail.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190527183508.9843-1-jrollins@caltech.edu \
--to=jrollins@caltech.edu \
--cc=notmuch@notmuchmail.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://yhetil.org/notmuch.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).