unofficial mirror of meta@public-inbox.org
 help / color / mirror / Atom feed
* [PATCH] disallow NUL characters in Message-ID and List-Id
@ 2023-11-27 22:20 Eric Wong
  2023-11-27 23:08 ` Eric W. Biederman
  0 siblings, 1 reply; 2+ messages in thread
From: Eric Wong @ 2023-11-27 22:20 UTC (permalink / raw)
  To: meta; +Cc: Eric W. Biederman

While MTAs seem to stop '\0' from appearing in headers, users
fetching archives via git remain susceptible to having '\0' land
in archives.  So we'll filter them out of Xapian and SQLite DBs
to avoid interopability problems with CLI tools since there's no
known messages in lore or any of my archives which feature them.

Avoiding '\0' will ensure all indexed Message-IDs and List-Ids
can be specified from the command-line (although some characters
will still require $(printf) contortions).

As with Message-ID, List-Id fields with /\n\t\r/ characters will
also be stripped for indexing.  I will assume whatever went wrong
with the References: header in
<https://public-inbox.org/git/656C30A1EFC89F6B2082D9B6@localhost/raw>
could also happen to the List-Id header.

This is inspired by commit aca47e05a6026c12c768753c87e6ff769ef6bee4
(Import: Don't copy nulls from emails into git, 2018-07-07)
---
 lib/PublicInbox/MID.pm       | 2 +-
 lib/PublicInbox/SearchIdx.pm | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/lib/PublicInbox/MID.pm b/lib/PublicInbox/MID.pm
index 97cf3a54..36c05855 100644
--- a/lib/PublicInbox/MID.pm
+++ b/lib/PublicInbox/MID.pm
@@ -115,7 +115,7 @@ sub uniq_mids ($;$) {
 	my @ret;
 	$seen ||= {};
 	foreach my $mid (@$mids) {
-		$mid =~ tr/\n\t\r//d;
+		$mid =~ tr/\n\t\r\0//d;
 		if (length($mid) > MAX_MID_SIZE) {
 			warn "Message-ID: <$mid> too long, truncating\n";
 			$mid = substr($mid, 0, MAX_MID_SIZE);
diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm
index 32598b7c..f569428c 100644
--- a/lib/PublicInbox/SearchIdx.pm
+++ b/lib/PublicInbox/SearchIdx.pm
@@ -414,6 +414,7 @@ sub index_list_id ($$$) {
 	for my $l ($hdr->header_raw('List-Id')) {
 		$l =~ /<([^>]+)>/ or next;
 		my $lid = lc $1;
+		$lid =~ tr/\n\t\r\0//d; # same rules as Message-ID
 		$doc->add_boolean_term('G' . $lid);
 		index_phrase($self, $lid, 1, 'XL'); # probabilistic
 	}

^ permalink raw reply related	[flat|nested] 2+ messages in thread

* Re: [PATCH] disallow NUL characters in Message-ID and List-Id
  2023-11-27 22:20 [PATCH] disallow NUL characters in Message-ID and List-Id Eric Wong
@ 2023-11-27 23:08 ` Eric W. Biederman
  0 siblings, 0 replies; 2+ messages in thread
From: Eric W. Biederman @ 2023-11-27 23:08 UTC (permalink / raw)
  To: Eric Wong; +Cc: meta

Eric Wong <e@80x24.org> writes:

> While MTAs seem to stop '\0' from appearing in headers, users
> fetching archives via git remain susceptible to having '\0' land
> in archives.  So we'll filter them out of Xapian and SQLite DBs
> to avoid interopability problems with CLI tools since there's no
> known messages in lore or any of my archives which feature them.
>
> Avoiding '\0' will ensure all indexed Message-IDs and List-Ids
> can be specified from the command-line (although some characters
> will still require $(printf) contortions).
>
> As with Message-ID, List-Id fields with /\n\t\r/ characters will
> also be stripped for indexing.  I will assume whatever went wrong
> with the References: header in
> <https://public-inbox.org/git/656C30A1EFC89F6B2082D9B6@localhost/raw>
> could also happen to the List-Id header.
>
> This is inspired by commit aca47e05a6026c12c768753c87e6ff769ef6bee4
> (Import: Don't copy nulls from emails into git, 2018-07-07)

That seems reasonable to me.

Eric


> ---
>  lib/PublicInbox/MID.pm       | 2 +-
>  lib/PublicInbox/SearchIdx.pm | 1 +
>  2 files changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/lib/PublicInbox/MID.pm b/lib/PublicInbox/MID.pm
> index 97cf3a54..36c05855 100644
> --- a/lib/PublicInbox/MID.pm
> +++ b/lib/PublicInbox/MID.pm
> @@ -115,7 +115,7 @@ sub uniq_mids ($;$) {
>  	my @ret;
>  	$seen ||= {};
>  	foreach my $mid (@$mids) {
> -		$mid =~ tr/\n\t\r//d;
> +		$mid =~ tr/\n\t\r\0//d;
>  		if (length($mid) > MAX_MID_SIZE) {
>  			warn "Message-ID: <$mid> too long, truncating\n";
>  			$mid = substr($mid, 0, MAX_MID_SIZE);
> diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm
> index 32598b7c..f569428c 100644
> --- a/lib/PublicInbox/SearchIdx.pm
> +++ b/lib/PublicInbox/SearchIdx.pm
> @@ -414,6 +414,7 @@ sub index_list_id ($$$) {
>  	for my $l ($hdr->header_raw('List-Id')) {
>  		$l =~ /<([^>]+)>/ or next;
>  		my $lid = lc $1;
> +		$lid =~ tr/\n\t\r\0//d; # same rules as Message-ID
>  		$doc->add_boolean_term('G' . $lid);
>  		index_phrase($self, $lid, 1, 'XL'); # probabilistic
>  	}

^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2023-11-27 23:08 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-11-27 22:20 [PATCH] disallow NUL characters in Message-ID and List-Id Eric Wong
2023-11-27 23:08 ` Eric W. Biederman

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).