angle brackets in "m:" and "refs:" in "lei q" JSON

angle brackets in "m:" and "refs:" in "lei q" JSON

[Eric Wong]

I'm thinking these shouldn't include angle brackets:

  "m": "<20210228122528.18552-2-e@80x24.org>",
  "refs": ["<20210228122528.18552-1-e@80x24.org>"],

Using angle brackets on the command-line requires quoting to
disambiguate against redirects, so it's a pain.  Leaving the
brackets in still works because of how Xapian's query parser
works, not because of anything we do on our end.

Since the actual headers are "Message-ID" and "References", (and
not "m" or "refs"), I think it's clear that we don't have to
match the raw mail contents exactly.  We RFC 2047 decode
"f|t|c|s" fields anyways instead of showing the raw values,
so more precedence for leaving out <>.

Re: angle brackets in "m:" and "refs:" in "lei q" JSON

[Kyle Meyer]

Eric Wong writes:

> I'm thinking these shouldn't include angle brackets:
>
>   "m": "<20210228122528.18552-2-e@80x24.org>",
>   "refs": ["<20210228122528.18552-1-e@80x24.org>"],
>
> Using angle brackets on the command-line requires quoting to
> disambiguate against redirects, so it's a pain.  Leaving the
> brackets in still works because of how Xapian's query parser
> works, not because of anything we do on our end.

I think it'd be nice to drop the brackets from a noise perspective too.

Also, does m: work with brackets?  Trying it out with a recent message
ID:

  $ lei q -q -I https://public-inbox.org/meta/ -f ldjson \
    '20210304203352.pd5mcg5pw4u2epzl@pengutronix.de'
  {"blob":"87304c8a8cae8ce400443b56309427aeee601505",...}

  $ lei q -q -I https://public-inbox.org/meta/ -f ldjson \
    m:'20210304203352.pd5mcg5pw4u2epzl@pengutronix.de'
  {"blob":"87304c8a8cae8ce400443b56309427aeee601505",...}

  $ lei q -q -I https://public-inbox.org/meta/ -f ldjson \
    '<20210304203352.pd5mcg5pw4u2epzl@pengutronix.de>'
  {"blob":"87304c8a8cae8ce400443b56309427aeee601505",...}

  $ lei q -q -I https://public-inbox.org/meta/ -f ldjson \
    m:'<20210304203352.pd5mcg5pw4u2epzl@pengutronix.de>'
  # no results

> Since the actual headers are "Message-ID" and "References", (and
> not "m" or "refs"), I think it's clear that we don't have to
> match the raw mail contents exactly.  We RFC 2047 decode
> "f|t|c|s" fields anyways instead of showing the raw values,
> so more precedence for leaving out <>.

Fwiw I don't think leaving out the brackets would be a source of
confusion.

[PATCH] lei q: remove angle brackets around Message-IDs

[Eric Wong]

Kyle Meyer <kyle@kyleam.com> wrote:
> Eric Wong writes:
> 
> > I'm thinking these shouldn't include angle brackets:
> >
> >   "m": "<20210228122528.18552-2-e@80x24.org>",
> >   "refs": ["<20210228122528.18552-1-e@80x24.org>"],
> >
> > Using angle brackets on the command-line requires quoting to
> > disambiguate against redirects, so it's a pain.  Leaving the
> > brackets in still works because of how Xapian's query parser
> > works, not because of anything we do on our end.
> 
> I think it'd be nice to drop the brackets from a noise perspective too.

Yes, we don't include them in name email address pairs, either.

> Also, does m: work with brackets?  Trying it out with a recent message
> ID:
> 
>   $ lei q -q -I https://public-inbox.org/meta/ -f ldjson \
>     '20210304203352.pd5mcg5pw4u2epzl@pengutronix.de'
>   {"blob":"87304c8a8cae8ce400443b56309427aeee601505",...}
> 
>   $ lei q -q -I https://public-inbox.org/meta/ -f ldjson \
>     m:'20210304203352.pd5mcg5pw4u2epzl@pengutronix.de'
>   {"blob":"87304c8a8cae8ce400443b56309427aeee601505",...}
> 
>   $ lei q -q -I https://public-inbox.org/meta/ -f ldjson \
>     '<20210304203352.pd5mcg5pw4u2epzl@pengutronix.de>'
>   {"blob":"87304c8a8cae8ce400443b56309427aeee601505",...}
> 
>   $ lei q -q -I https://public-inbox.org/meta/ -f ldjson \
>     m:'<20210304203352.pd5mcg5pw4u2epzl@pengutronix.de>'
>   # no results

Odd, I'm not sure about that one...  It's probably something
the Xapian query parser is doing internally and nothing on
our end...

> > Since the actual headers are "Message-ID" and "References", (and
> > not "m" or "refs"), I think it's clear that we don't have to
> > match the raw mail contents exactly.  We RFC 2047 decode
> > "f|t|c|s" fields anyways instead of showing the raw values,
> > so more precedence for leaving out <>.
> 
> Fwiw I don't think leaving out the brackets would be a source of
> confusion.

Agreed.  And I'm now wondering if we should start indexing
"References:" to be a searchable header with the "refs:" prefix
(but also wary about increasing disk space usage as a result...)

In any case, this denoises the output a bit:
------------8<----------
Subject: [PATCH] lei q: remove angle brackets around Message-IDs

They're unnecessary visual noise, and angle brackets don't
always work as intended when going through Xapian's query
parser.

Since we already use "m:" and "refs:" instead of the actual
header names, it should be obvious we're at liberty to
abbreviate such things

Link: https://public-inbox.org/meta/20210304184348.GA19350@dcvr/
---
 lib/PublicInbox/LeiOverview.pm | 5 ++---
 t/lei-externals.t              | 2 +-
 t/lei-q-thread.t               | 8 ++++----
 xt/net_writer-imap.t           | 2 +-
 4 files changed, 8 insertions(+), 9 deletions(-)

diff --git a/lib/PublicInbox/LeiOverview.pm b/lib/PublicInbox/LeiOverview.pm
index 4db1d8c8..01556273 100644
--- a/lib/PublicInbox/LeiOverview.pm
+++ b/lib/PublicInbox/LeiOverview.pm
@@ -141,17 +141,16 @@ sub _unbless_smsg {
 	$smsg->{dt} = _iso8601(delete $smsg->{ds}); # JMAP UTCDate
 	$smsg->{pct} = get_pct($mitem) if $mitem;
 	if (my $r = delete $smsg->{references}) {
-		$smsg->{refs} = [ map { "<$_>" } ($r =~ m/$MID_EXTRACT/go) ];
+		$smsg->{refs} = [ map { $_ } ($r =~ m/$MID_EXTRACT/go) ];
 	}
 	if (my $m = delete($smsg->{mid})) {
-		$smsg->{'m'} = "<$m>";
+		$smsg->{'m'} = $m;
 	}
 	for my $f (qw(from to cc)) {
 		my $v = delete $smsg->{$f} or next;
 		$smsg->{substr($f, 0, 1)} = pairs($v);
 	}
 	$smsg->{'s'} = delete $smsg->{subject};
-	# can we be bothered to parse From/To/Cc into arrays?
 	scalar { %$smsg }; # unbless
 }
 
diff --git a/t/lei-externals.t b/t/lei-externals.t
index 29667640..2a92d101 100644
--- a/t/lei-externals.t
+++ b/t/lei-externals.t
@@ -25,7 +25,7 @@ SKIP: {
 	lei_ok(@cmd, \"query $url");
 	is($lei_err, '', "no errors on $url");
 	my $res = json_utf8->decode($lei_out);
-	is($res->[0]->{'m'}, "<$mid>", "got expected mid from $url") or
+	is($res->[0]->{'m'}, $mid, "got expected mid from $url") or
 		skip 'further remote tests', 1;
 	lei_ok(@cmd, 'd:..20101002', \'no results, no error');
 	is($lei_err, '', 'no output on 404, matching local FS behavior');
diff --git a/t/lei-q-thread.t b/t/lei-q-thread.t
index 28c639f5..e24fb2cb 100644
--- a/t/lei-q-thread.t
+++ b/t/lei-q-thread.t
@@ -27,9 +27,9 @@ test_lei(sub {
 	is(scalar(@$res), 3, 'got 2 results');
 	pop @$res;
 	my %m = map { $_->{'m'} => $_ } @$res;
-	is_deeply($m{'<testmessage@example.com>'}->{kw}, ['seen'],
+	is_deeply($m{'testmessage@example.com'}->{kw}, ['seen'],
 		'flag set in direct hit');
-	'TODO' or is_deeply($m{'<a-reply@miss>'}->{kw}, ['draft'],
+	'TODO' or is_deeply($m{'a-reply@miss'}->{kw}, ['draft'],
 		'flag set in thread hit');
 
 	lei_ok qw(q -t -t m:testmessage@example.com);
@@ -37,9 +37,9 @@ test_lei(sub {
 	is(scalar(@$res), 3, 'got 2 results with -t -t');
 	pop @$res;
 	%m = map { $_->{'m'} => $_ } @$res;
-	is_deeply($m{'<testmessage@example.com>'}->{kw}, ['flagged', 'seen'],
+	is_deeply($m{'testmessage@example.com'}->{kw}, ['flagged', 'seen'],
 		'flagged set in direct hit');
-	'TODO' or is_deeply($m{'<testmessage@example.com>'}->{kw}, ['draft'],
+	'TODO' or is_deeply($m{'testmessage@example.com'}->{kw}, ['draft'],
 		'flagged set in direct hit');
 	lei_ok qw(q -tt m:testmessage@example.com --only), "$ro_home/t2";
 	$res = json_utf8->decode($lei_out);
diff --git a/xt/net_writer-imap.t b/xt/net_writer-imap.t
index c24fa993..3631d932 100644
--- a/xt/net_writer-imap.t
+++ b/xt/net_writer-imap.t
@@ -168,7 +168,7 @@ test_lei(sub {
 	is_deeply($empty, [], 'clobbered folder');
 	lei_ok qw(q -o /dev/stdout m:testmessage@example.com --no-external);
 	$res = json_utf8->decode($lei_out)->[0];
-	is_deeply([@$res{qw(m kw)}], ['<testmessage@example.com>', ['seen']],
+	is_deeply([@$res{qw(m kw)}], ['testmessage@example.com', ['seen']],
 		'kw set');
 });