[PATCH] disallow NUL characters in Message-ID and List-Id

[PATCH] disallow NUL characters in Message-ID and List-Id

[Eric Wong]

While MTAs seem to stop '\0' from appearing in headers, users
fetching archives via git remain susceptible to having '\0' land
in archives.  So we'll filter them out of Xapian and SQLite DBs
to avoid interopability problems with CLI tools since there's no
known messages in lore or any of my archives which feature them.

Avoiding '\0' will ensure all indexed Message-IDs and List-Ids
can be specified from the command-line (although some characters
will still require $(printf) contortions).

As with Message-ID, List-Id fields with /\n\t\r/ characters will
also be stripped for indexing.  I will assume whatever went wrong
with the References: header in
<https://public-inbox.org/git/656C30A1EFC89F6B2082D9B6@localhost/raw>
could also happen to the List-Id header.

This is inspired by commit aca47e05a6026c12c768753c87e6ff769ef6bee4
(Import: Don't copy nulls from emails into git, 2018-07-07)
---
 lib/PublicInbox/MID.pm       | 2 +-
 lib/PublicInbox/SearchIdx.pm | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/lib/PublicInbox/MID.pm b/lib/PublicInbox/MID.pm
index 97cf3a54..36c05855 100644
--- a/lib/PublicInbox/MID.pm
+++ b/lib/PublicInbox/MID.pm
@@ -115,7 +115,7 @@ sub uniq_mids ($;$) {
 	my @ret;
 	$seen ||= {};
 	foreach my $mid (@$mids) {
-		$mid =~ tr/\n\t\r//d;
+		$mid =~ tr/\n\t\r\0//d;
 		if (length($mid) > MAX_MID_SIZE) {
 			warn "Message-ID: <$mid> too long, truncating\n";
 			$mid = substr($mid, 0, MAX_MID_SIZE);
diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm
index 32598b7c..f569428c 100644
--- a/lib/PublicInbox/SearchIdx.pm
+++ b/lib/PublicInbox/SearchIdx.pm
@@ -414,6 +414,7 @@ sub index_list_id ($$$) {
 	for my $l ($hdr->header_raw('List-Id')) {
 		$l =~ /<([^>]+)>/ or next;
 		my $lid = lc $1;
+		$lid =~ tr/\n\t\r\0//d; # same rules as Message-ID
 		$doc->add_boolean_term('G' . $lid);
 		index_phrase($self, $lid, 1, 'XL'); # probabilistic
 	}

Re: [PATCH] disallow NUL characters in Message-ID and List-Id

[Eric W. Biederman]

Eric Wong <e@80x24.org> writes:

> While MTAs seem to stop '\0' from appearing in headers, users
> fetching archives via git remain susceptible to having '\0' land
> in archives.  So we'll filter them out of Xapian and SQLite DBs
> to avoid interopability problems with CLI tools since there's no
> known messages in lore or any of my archives which feature them.
>
> Avoiding '\0' will ensure all indexed Message-IDs and List-Ids
> can be specified from the command-line (although some characters
> will still require $(printf) contortions).
>
> As with Message-ID, List-Id fields with /\n\t\r/ characters will
> also be stripped for indexing.  I will assume whatever went wrong
> with the References: header in
> <https://public-inbox.org/git/656C30A1EFC89F6B2082D9B6@localhost/raw>
> could also happen to the List-Id header.
>
> This is inspired by commit aca47e05a6026c12c768753c87e6ff769ef6bee4
> (Import: Don't copy nulls from emails into git, 2018-07-07)

That seems reasonable to me.

Eric


> ---
>  lib/PublicInbox/MID.pm       | 2 +-
>  lib/PublicInbox/SearchIdx.pm | 1 +
>  2 files changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/lib/PublicInbox/MID.pm b/lib/PublicInbox/MID.pm
> index 97cf3a54..36c05855 100644
> --- a/lib/PublicInbox/MID.pm
> +++ b/lib/PublicInbox/MID.pm
> @@ -115,7 +115,7 @@ sub uniq_mids ($;$) {
>  	my @ret;
>  	$seen ||= {};
>  	foreach my $mid (@$mids) {
> -		$mid =~ tr/\n\t\r//d;
> +		$mid =~ tr/\n\t\r\0//d;
>  		if (length($mid) > MAX_MID_SIZE) {
>  			warn "Message-ID: <$mid> too long, truncating\n";
>  			$mid = substr($mid, 0, MAX_MID_SIZE);
> diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm
> index 32598b7c..f569428c 100644
> --- a/lib/PublicInbox/SearchIdx.pm
> +++ b/lib/PublicInbox/SearchIdx.pm
> @@ -414,6 +414,7 @@ sub index_list_id ($$$) {
>  	for my $l ($hdr->header_raw('List-Id')) {
>  		$l =~ /<([^>]+)>/ or next;
>  		my $lid = lc $1;
> +		$lid =~ tr/\n\t\r\0//d; # same rules as Message-ID
>  		$doc->add_boolean_term('G' . $lid);
>  		index_phrase($self, $lid, 1, 'XL'); # probabilistic
>  	}