unofficial mirror of meta@public-inbox.org
 help / color / mirror / Atom feed
* [PATCH 0/8] lei NNTP/IMAP .onion support and misc fixes
@ 2021-04-30  9:24 Eric Wong
  2021-04-30  9:24 ` [PATCH 1/8] lei sucks: preserve utsname.machine, add "x86" where appropriate Eric Wong
                   ` (7 more replies)
  0 siblings, 8 replies; 9+ messages in thread
From: Eric Wong @ 2021-04-30  9:24 UTC (permalink / raw)
  To: meta

Attempting to use torsocks(1) for NNTP or IMAP could get tricky.
Fortunately, IO::Socket::Socks is packaged for on CentOS 7,
FreeBSD, and Debian, so it seems to be a reasonable way to
support NNTP and IMAP Tor onions.

--proxy= (shared with curl) is supported for one-off
command-line use, but imap.proxy and nntp.proxy are both
supported along with URL-matching variants with git 1.8.5 (or
git 2.26 for wildcard URL matching).

Only socks5h:// proxies are supported (the default with
IO::Socket::Socks), which is what Tor uses.  I doubt its worth
the effort (and potential for DNS request leaks) to support
prior versions of SOCKS in 2021.

Eric Wong (8):
  lei sucks: preserve utsname.machine, add "x86" where appropriate
  lei_curl: improve correctness of LD_PRELOAD check
  lei: kill old PIDs when dropping
  lei: ensure autoflush(1) is on STDERR
  net_reader: {nn,mic}_for: use prototypes for internal subs
  lei: IMAP .onion support via --proxy=s switch
  net_reader: Net::NNTP --proxy=socks5h:// support
  net_reader: support (imap|nntp).proxy in config file

 MANIFEST                        |  2 +
 lib/PublicInbox/Config.pm       |  1 +
 lib/PublicInbox/LEI.pm          | 24 ++++++++---
 lib/PublicInbox/LeiCurl.pm      |  2 +-
 lib/PublicInbox/LeiInput.pm     |  2 +-
 lib/PublicInbox/LeiSucks.pm     |  3 +-
 lib/PublicInbox/LeiToMail.pm    |  4 +-
 lib/PublicInbox/NetNNTPSocks.pm | 33 +++++++++++++++
 lib/PublicInbox/NetReader.pm    | 72 +++++++++++++++++++++++++++------
 xt/net_nntp_socks.t             | 22 ++++++++++
 10 files changed, 141 insertions(+), 24 deletions(-)
 create mode 100644 lib/PublicInbox/NetNNTPSocks.pm
 create mode 100644 xt/net_nntp_socks.t

^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH 1/8] lei sucks: preserve utsname.machine, add "x86" where appropriate
  2021-04-30  9:24 [PATCH 0/8] lei NNTP/IMAP .onion support and misc fixes Eric Wong
@ 2021-04-30  9:24 ` Eric Wong
  2021-04-30  9:24 ` [PATCH 2/8] lei_curl: improve correctness of LD_PRELOAD check Eric Wong
                   ` (6 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-04-30  9:24 UTC (permalink / raw)
  To: meta

It's helpful for us to distinguish x86 kernels from x86_64
kernels when using an x86 userspace.  OSes are dropping i386
support and only support i486 and newer, so "x86" is a more
appropriate description for that platform than "i386".
---
 lib/PublicInbox/LeiSucks.pm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lib/PublicInbox/LeiSucks.pm b/lib/PublicInbox/LeiSucks.pm
index d364a856..2ce64d62 100644
--- a/lib/PublicInbox/LeiSucks.pm
+++ b/lib/PublicInbox/LeiSucks.pm
@@ -18,7 +18,8 @@ sub lei_sucks {
 	$lei->start_pager if -t $lei->{1};
 	my ($os, undef, $rel, undef, $mac)= POSIX::uname();
 	if ($mac eq 'x86_64' && $Config{ptrsize} == 4) {
-		$mac = $Config{cppsymbols} =~ /\b__ILP32__=1\b/ ? 'x32' : 'i386'
+		$mac .= $Config{cppsymbols} =~ /\b__ILP32__=1\b/ ?
+			',u=x32' : ',u=x86';
 	}
 	eval { require PublicInbox };
 	my $pi_ver = eval('$PublicInbox::VERSION') // '(???)';

^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [PATCH 2/8] lei_curl: improve correctness of LD_PRELOAD check
  2021-04-30  9:24 [PATCH 0/8] lei NNTP/IMAP .onion support and misc fixes Eric Wong
  2021-04-30  9:24 ` [PATCH 1/8] lei sucks: preserve utsname.machine, add "x86" where appropriate Eric Wong
@ 2021-04-30  9:24 ` Eric Wong
  2021-04-30  9:24 ` [PATCH 3/8] lei: kill old PIDs when dropping Eric Wong
                   ` (5 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-04-30  9:24 UTC (permalink / raw)
  To: meta

LD_PRELOAD sent by a client can't affect lei-daemon.
---
 lib/PublicInbox/Config.pm  | 1 +
 lib/PublicInbox/LeiCurl.pm | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/lib/PublicInbox/Config.pm b/lib/PublicInbox/Config.pm
index 016f50ec..3f0f5a01 100644
--- a/lib/PublicInbox/Config.pm
+++ b/lib/PublicInbox/Config.pm
@@ -12,6 +12,7 @@ use strict;
 use v5.10.1;
 use PublicInbox::Inbox;
 use PublicInbox::Spawn qw(popen_rd);
+our $LD_PRELOAD = $ENV{LD_PRELOAD}; # only valid at startup
 
 sub _array ($) { ref($_[0]) eq 'ARRAY' ? $_[0] : [ $_[0] ] }
 
diff --git a/lib/PublicInbox/LeiCurl.pm b/lib/PublicInbox/LeiCurl.pm
index 69c64cdf..ce57e796 100644
--- a/lib/PublicInbox/LeiCurl.pm
+++ b/lib/PublicInbox/LeiCurl.pm
@@ -55,7 +55,7 @@ sub torsocks { # useful for "git clone" and "git fetch", too
 	$opt->{torsocks} = 'false' if $opt->{'no-torsocks'};
 	my $torsocks = $opt->{torsocks} //= 'auto';
 	if ($torsocks eq 'auto' && substr($uri->host, -6) eq '.onion' &&
-			(($lei->{env}->{LD_PRELOAD}//'') !~ /torsocks/)) {
+		($PublicInbox::Config::LD_PRELOAD//'') !~ m!/libtorsocks\b!) {
 		# "auto" continues anyways if torsocks is missing;
 		# a proxy may be specified via CLI, curlrc,
 		# environment variable, or even firewall rule

^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [PATCH 3/8] lei: kill old PIDs when dropping
  2021-04-30  9:24 [PATCH 0/8] lei NNTP/IMAP .onion support and misc fixes Eric Wong
  2021-04-30  9:24 ` [PATCH 1/8] lei sucks: preserve utsname.machine, add "x86" where appropriate Eric Wong
  2021-04-30  9:24 ` [PATCH 2/8] lei_curl: improve correctness of LD_PRELOAD check Eric Wong
@ 2021-04-30  9:24 ` Eric Wong
  2021-04-30  9:24 ` [PATCH 4/8] lei: ensure autoflush(1) is on STDERR Eric Wong
                   ` (4 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-04-30  9:24 UTC (permalink / raw)
  To: meta

This ensures hitting Ctrl-C on a long-running "lei convert" or
similar will stop the WQ worker, even after we've closed
the WQ socketpair in the daemon.
---
 lib/PublicInbox/LEI.pm | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 52ce8ec2..3468094f 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -387,7 +387,14 @@ my @WQ_KEYS = qw(lxs l2m wq1); # internal workers
 
 sub _drop_wq {
 	my ($self) = @_;
-	for my $wq (grep(defined, delete(@$self{@WQ_KEYS}))) { $wq->DESTROY }
+	for my $wq (grep(defined, delete(@$self{@WQ_KEYS}))) {
+		if ($wq->wq_kill) {
+			$wq->wq_close(0, undef, $self);
+		} elsif ($wq->wq_kill_old) {
+			$wq->wq_wait_old(undef, $self);
+		}
+		$wq->DESTROY;
+	}
 }
 
 # pronounced "exit": x_it(1 << 8) => exit(1); x_it(13) => SIGPIPE

^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [PATCH 4/8] lei: ensure autoflush(1) is on STDERR
  2021-04-30  9:24 [PATCH 0/8] lei NNTP/IMAP .onion support and misc fixes Eric Wong
                   ` (2 preceding siblings ...)
  2021-04-30  9:24 ` [PATCH 3/8] lei: kill old PIDs when dropping Eric Wong
@ 2021-04-30  9:24 ` Eric Wong
  2021-04-30  9:24 ` [PATCH 5/8] net_reader: {nn,mic}_for: use prototypes for internal subs Eric Wong
                   ` (3 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-04-30  9:24 UTC (permalink / raw)
  To: meta

This fixes error reporting for oneshot tests in xt/lei-auth-failure.t
---
 lib/PublicInbox/LEI.pm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 3468094f..6a82d497 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -500,6 +500,7 @@ sub _lei_atfork_child {
 		}
 	} else { # worker, Net::NNTP (Net::Cmd) uses STDERR directly
 		open STDERR, '+>&='.fileno($self->{2}) or warn "open $!";
+		STDERR->autoflush(1);
 	}
 	close($_) for (grep(defined, delete @$self{qw(3 old_1 au_done)}));
 	if (my $op_c = delete $self->{pkt_op_c}) {
@@ -676,6 +677,7 @@ sub lazy_cb ($$$) {
 sub dispatch {
 	my ($self, $cmd, @argv) = @_;
 	local $current_lei = $self; # for __WARN__
+	$self->{2}->autoflush(1); # keep stdout buffered until x_it|DESTROY
 	dump_and_clear_log("from previous run\n");
 	return _help($self, 'no command given') unless defined($cmd);
 	# do not support Getopt bundling for this
@@ -1006,7 +1008,6 @@ sub accept_dispatch { # Listener {post_accept} callback
 		}
 		$i == 4 or return send($sock, 'not enough FDs='.($i-1), MSG_EOR)
 	}
-	$self->{2}->autoflush(1); # keep stdout buffered until x_it|DESTROY
 	# $ENV_STR = join('', map { "\0$_=$ENV{$_}" } keys %ENV);
 	# $buf = "$argc\0".join("\0", @ARGV).$ENV_STR."\0\0";
 	substr($buf, -2, 2, '') eq "\0\0" or  # s/\0\0\z//

^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [PATCH 5/8] net_reader: {nn,mic}_for: use prototypes for internal subs
  2021-04-30  9:24 [PATCH 0/8] lei NNTP/IMAP .onion support and misc fixes Eric Wong
                   ` (3 preceding siblings ...)
  2021-04-30  9:24 ` [PATCH 4/8] lei: ensure autoflush(1) is on STDERR Eric Wong
@ 2021-04-30  9:24 ` Eric Wong
  2021-04-30  9:24 ` [PATCH 6/8] lei: IMAP .onion support via --proxy=s switch Eric Wong
                   ` (2 subsequent siblings)
  7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-04-30  9:24 UTC (permalink / raw)
  To: meta

We don't use these subs elsewhere, so stick prototypes on them
to give them a little extra checking.
---
 lib/PublicInbox/NetReader.pm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index 3fc37b10..b9365c05 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -27,7 +27,7 @@ sub uri_section ($) {
 sub auth_anon_cb { '' }; # for Mail::IMAPClient::Authcallback
 
 # mic_for may prompt the user and store auth info, prepares mic_get
-sub mic_for { # mic = Mail::IMAPClient
+sub mic_for ($$$$) { # mic = Mail::IMAPClient
 	my ($self, $url, $mic_args, $lei) = @_;
 	require PublicInbox::URIimap;
 	my $uri = PublicInbox::URIimap->new($url);
@@ -133,7 +133,7 @@ E: <$uri> STARTTLS requested and failed
 	$nn;
 }
 
-sub nn_for ($$$;$) { # nn = Net::NNTP
+sub nn_for ($$$$) { # nn = Net::NNTP
 	my ($self, $uri, $nn_args, $lei) = @_;
 	my $sec = uri_section($uri);
 	my $nntp_opt = $self->{nntp_opt}->{$sec} //= {};

^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [PATCH 6/8] lei: IMAP .onion support via --proxy=s switch
  2021-04-30  9:24 [PATCH 0/8] lei NNTP/IMAP .onion support and misc fixes Eric Wong
                   ` (4 preceding siblings ...)
  2021-04-30  9:24 ` [PATCH 5/8] net_reader: {nn,mic}_for: use prototypes for internal subs Eric Wong
@ 2021-04-30  9:24 ` Eric Wong
  2021-04-30  9:24 ` [PATCH 7/8] net_reader: Net::NNTP --proxy=socks5h:// support Eric Wong
  2021-04-30  9:24 ` [PATCH 8/8] net_reader: support (imap|nntp).proxy in config file Eric Wong
  7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-04-30  9:24 UTC (permalink / raw)
  To: meta

Mail::IMAPClient provides the ability to pass a pre-connected
Socket to it.  We can rely on this functionality to use
IO::Socket::Socks in place whatever socket class
Mail::IMAPClient chooses to use.

The --proxy=s is shared with curl(1), though we only support
socks5h:// at the moment.  Is there any need for SOCKS4 or SOCKS5
without name resolution?  Tor .onions require socks5h:// for
name resolution and to prevent data leakage.
---
 lib/PublicInbox/LEI.pm       | 12 ++++++++----
 lib/PublicInbox/LeiInput.pm  |  2 +-
 lib/PublicInbox/LeiToMail.pm |  4 ++--
 lib/PublicInbox/NetReader.pm | 31 ++++++++++++++++++++++++++++---
 4 files changed, 39 insertions(+), 10 deletions(-)

diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 6a82d497..bb67fc0b 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -188,7 +188,8 @@ our %CMD = ( # sorted in order of importance/use:
 	qw(stdin| threads|t from|f=s mid=s oid=s), @c_opt ],
 'tag' => [ 'KEYWORDS...',
 	'set/unset keywords and/or labels on message(s)',
-	qw(stdin| in-format|F=s input|i=s@ oid=s@ mid=s@), @c_opt,
+	qw(stdin| in-format|F=s input|i=s@ oid=s@ mid=s@),
+	qw(no-torsocks torsocks=s), PublicInbox::LeiQuery::curl_opt(), @c_opt,
 	pass_through('-kw:foo for delete') ],
 'forget' => [ '[--stdin|--oid=OID|--by-mid=MID]',
 	"exclude message(s) on stdin from `q' search results",
@@ -211,11 +212,12 @@ our %CMD = ( # sorted in order of importance/use:
 'import' => [ 'LOCATION...|--stdin',
 	'one-time import/update from URL or filesystem',
 	qw(stdin| offset=i recursive|r exclude=s include|I=s
-	lock=s@ in-format|F=s kw! verbose|v+ incremental! mail-sync!), @c_opt ],
+	lock=s@ in-format|F=s kw! verbose|v+ incremental! mail-sync!),
+	qw(no-torsocks torsocks=s), PublicInbox::LeiQuery::curl_opt(), @c_opt ],
 'convert' => [ 'LOCATION...|--stdin',
 	'one-time conversion from URL or filesystem to another format',
-	qw(stdin| in-format|F=s out-format|f=s output|mfolder|o=s
-	lock=s@ kw!), @c_opt ],
+	qw(stdin| in-format|F=s out-format|f=s output|mfolder|o=s lock=s@ kw!),
+	qw(no-torsocks torsocks=s), PublicInbox::LeiQuery::curl_opt(), @c_opt ],
 'p2q' => [ 'FILE|COMMIT_OID|--stdin',
 	"use a patch to generate a query for `lei q --stdin'",
 	qw(stdin| want|w=s@ uri debug), @c_opt ],
@@ -277,6 +279,8 @@ my %OPTDESC = (
 'path-a|a=s' => 'pre-image pathname associated with OID',
 'path-b|b=s' => 'post-image pathname associated with OID',
 'git-dir=s@' => 'additional git repository to scan',
+'proxy=s' => [ 'PROTO://HOST[:PORT]', # shared with curl(1)
+	"proxy for (e.g. `socks5h://0:9050')" ],
 'torsocks=s' => ['VAL|auto|no|yes',
 		'whether or not to wrap git and curl commands with torsocks'],
 'no-torsocks' => 'alias for --torsocks=no',
diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm
index 277ad88d..86f300c3 100644
--- a/lib/PublicInbox/LeiInput.pm
+++ b/lib/PublicInbox/LeiInput.pm
@@ -294,7 +294,7 @@ $input is `eml', not --in-format=$in_fmt
 	}
 	if ($net) {
 		$net->{-can_die} = 1;
-		if (my $err = $net->errors) {
+		if (my $err = $net->errors($lei)) {
 			return $lei->fail($err);
 		}
 		$net->{quiet} = $lei->{opt}->{quiet};
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index fa3af710..eda4701c 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -351,14 +351,14 @@ sub new {
 		require PublicInbox::MboxReader;
 		$self->can("eml2$fmt") or die "bad mbox format: $fmt\n";
 		$self->{base_type} = 'mbox';
-	} elsif ($fmt =~ /\Aimaps?\z/) { # TODO .onion support
+	} elsif ($fmt =~ /\Aimaps?\z/) {
 		require PublicInbox::NetWriter;
 		require PublicInbox::URIimap;
 		my $net = PublicInbox::NetWriter->new;
 		$net->{quiet} = $lei->{opt}->{quiet};
 		my $uri = PublicInbox::URIimap->new($dst)->canonical;
 		$net->add_url($uri);
-		my $err = $net->errors;
+		my $err = $net->errors($lei);
 		return $lei->fail($err) if $err;
 		$uri->mailbox or return $lei->fail("No mailbox: $dst");
 		$self->{uri} = $uri;
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index b9365c05..ac23e701 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -7,6 +7,7 @@ use strict;
 use v5.10.1;
 use parent qw(Exporter PublicInbox::IPC);
 use PublicInbox::Eml;
+use PublicInbox::Config;
 our %IMAPflags2kw = map {; "\\\u$_" => $_ } qw(seen answered flagged draft);
 $IMAPflags2kw{'$Forwarded'} = 'forwarded';  # RFC 5550
 
@@ -51,7 +52,16 @@ sub mic_for ($$$$) { # mic = Mail::IMAPClient
 		%$common, # may set Starttls, Compress, Debug ....
 	};
 	require PublicInbox::IMAPClient;
-	my $mic = PublicInbox::IMAPClient->new(%$mic_arg) or
+	my %socks;
+	if ($lei && $lei->{socks5h}) {
+		my %opt = %{$lei->{socks5h}};
+		$opt{ConnectAddr} = delete $mic_arg->{Server};
+		$opt{ConnectPort} = delete $mic_arg->{Port};
+		$socks{Socket} = IO::Socket::Socks->new(%opt) or die
+			"E: <$url> ".eval('$IO::Socket::Socks::SOCKS_ERROR');
+		$self->{mic_socks5h} = \%opt;
+	}
+	my $mic = PublicInbox::IMAPClient->new(%$mic_arg, %socks) or
 		die "E: <$url> new: $@\n";
 
 	# default to using STARTTLS if it's available, but allow
@@ -331,7 +341,7 @@ sub add_url {
 }
 
 sub errors {
-	my ($self) = @_;
+	my ($self, $lei) = @_;
 	if (my $u = $self->{unsupported_url}) {
 		return "Unsupported URL(s): @$u";
 	}
@@ -343,6 +353,16 @@ sub errors {
 		eval { require Net::NNTP } or
 			die "Net::NNTP is required for NNTP:\n$@\n";
 	}
+	if ($lei && (($lei->{opt}->{proxy}//'') =~ m!\Asocks5h://
+				(?: \[ ([^\]]+) \] | ([^:/]+) )
+				(?::([0-9]+))?/?(?:,|\z)!ix)) {
+		my ($h, $p) = ($1 // $2, $3 + 0);
+		$h = '127.0.0.1' if $h eq '0';
+		eval { require IO::Socket::Socks } or die <<EOM;
+IO::Socket::Socks missing for socks5h://$h:$p
+EOM
+		$lei->{socks5h} = { ProxyAddr => $h, ProxyPort => $p };
+	}
 	undef;
 }
 
@@ -507,7 +527,12 @@ sub mic_get {
 			$mic_arg->{Authcallback} = $self->can($cb_name);
 		}
 	}
-	my $mic = PublicInbox::IMAPClient->new(%$mic_arg);
+	my %socks;
+	if (my $s5h = $self->{mic_socks5h}) {
+		$socks{Socket} = IO::Socket::Socks->new(%$s5h) or die
+			"E: <$$uri> ".eval('$IO::Socket::Socks::SOCKS_ERROR');
+	}
+	my $mic = PublicInbox::IMAPClient->new(%$mic_arg, %socks);
 	$cached //= {}; # invalid placeholder if no cache enabled
 	$mic && $mic->IsConnected ? ($cached->{$sec} = $mic) : undef;
 }

^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [PATCH 7/8] net_reader: Net::NNTP --proxy=socks5h:// support
  2021-04-30  9:24 [PATCH 0/8] lei NNTP/IMAP .onion support and misc fixes Eric Wong
                   ` (5 preceding siblings ...)
  2021-04-30  9:24 ` [PATCH 6/8] lei: IMAP .onion support via --proxy=s switch Eric Wong
@ 2021-04-30  9:24 ` Eric Wong
  2021-04-30  9:24 ` [PATCH 8/8] net_reader: support (imap|nntp).proxy in config file Eric Wong
  7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-04-30  9:24 UTC (permalink / raw)
  To: meta

Since Net::NNTP doesn't support Socket or RawSocket
options/accessors like Mail::IMAPClient does; we must perform
localized @ISA manipulation and massage Net::NNTP into using
IO::Socket::Socks rather than IO::Socket::IP.

This is a bit fragile, but Net::Cmd and Net::NNTP rarely change;
and I keep an eye on them, anyways.
---
 MANIFEST                        |  2 ++
 lib/PublicInbox/NetNNTPSocks.pm | 33 +++++++++++++++++++++++++++++++++
 lib/PublicInbox/NetReader.pm    | 12 +++++++++++-
 xt/net_nntp_socks.t             | 22 ++++++++++++++++++++++
 4 files changed, 68 insertions(+), 1 deletion(-)
 create mode 100644 lib/PublicInbox/NetNNTPSocks.pm
 create mode 100644 xt/net_nntp_socks.t

diff --git a/MANIFEST b/MANIFEST
index 5933ddf4..bc2ad671 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -241,6 +241,7 @@ lib/PublicInbox/NDC_PP.pm
 lib/PublicInbox/NNTP.pm
 lib/PublicInbox/NNTPD.pm
 lib/PublicInbox/NNTPdeflate.pm
+lib/PublicInbox/NetNNTPSocks.pm
 lib/PublicInbox/NetReader.pm
 lib/PublicInbox/NetWriter.pm
 lib/PublicInbox/NewsWWW.pm
@@ -518,6 +519,7 @@ xt/lei-auth-fail.t
 xt/mem-imapd-tls.t
 xt/mem-msgview.t
 xt/msgtime_cmp.t
+xt/net_nntp_socks.t
 xt/net_writer-imap.t
 xt/nntpd-validate.t
 xt/perf-msgview.t
diff --git a/lib/PublicInbox/NetNNTPSocks.pm b/lib/PublicInbox/NetNNTPSocks.pm
new file mode 100644
index 00000000..8495204a
--- /dev/null
+++ b/lib/PublicInbox/NetNNTPSocks.pm
@@ -0,0 +1,33 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# wrap Net::NNTP client with SOCKS support
+package PublicInbox::NetNNTPSocks;
+use strict;
+use v5.10.1;
+use Net::NNTP;
+our %OPT;
+our @ISA = qw(IO::Socket::Socks);
+my @SOCKS_KEYS = qw(ProxyAddr ProxyPort SocksVersion SocksDebug SocksResolve);
+
+# use this instead of Net::NNTP->new if using Proxy*
+sub new_socks {
+	my (undef, %opt) = @_;
+	require IO::Socket::Socks;
+	local @Net::NNTP::ISA = (qw(Net::Cmd), __PACKAGE__);
+	local %OPT = map {;
+		defined($opt{$_}) ? ($_ => $opt{$_}) : ()
+	} @SOCKS_KEYS;
+	Net::NNTP->new(%opt); # this calls our new() below:
+}
+
+# called by Net::NNTP->new
+sub new {
+	my ($self, %opt) = @_;
+	@OPT{qw(ConnectAddr ConnectPort)} = @opt{qw(PeerAddr PeerPort)};
+	my $ret = $self->SUPER::new(%OPT) or
+		die 'SOCKS error: '.eval('$IO::Socket::Socks::SOCKS_ERROR');
+	$ret;
+}
+
+1;
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index ac23e701..b2c4fee2 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -116,7 +116,13 @@ sub try_starttls ($) {
 
 sub nn_new ($$$) {
 	my ($nn_arg, $nntp_opt, $uri) = @_;
-	my $nn = Net::NNTP->new(%$nn_arg) or die "E: <$uri> new: $!\n";
+	my $nn;
+	if (defined $nn_arg->{ProxyAddr}) {
+		eval { $nn = PublicInbox::NetNNTPSocks->new_socks(%$nn_arg) };
+		die "E: <$uri> $@\n" if $@;
+	} else {
+		$nn = Net::NNTP->new(%$nn_arg) or die "E: <$uri> new: $!\n";
+	}
 
 	# default to using STARTTLS if it's available, but allow
 	# it to be disabled for localhost/VPN users
@@ -170,6 +176,10 @@ sub nn_for ($$$$) { # nn = Net::NNTP
 		SSL => $uri->secure, # snews == nntps
 		%$common, # may Debug ....
 	};
+	if ($lei && $lei->{socks5h}) {
+		require PublicInbox::NetNNTPSocks;
+		%$nn_arg = (%$nn_arg, %{$lei->{socks5h}});
+	}
 	my $nn = nn_new($nn_arg, $nntp_opt, $uri);
 	if ($cred) {
 		$cred->fill($lei); # may prompt user here
diff --git a/xt/net_nntp_socks.t b/xt/net_nntp_socks.t
new file mode 100644
index 00000000..4a144fd8
--- /dev/null
+++ b/xt/net_nntp_socks.t
@@ -0,0 +1,22 @@
+#!perl -w
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+use v5.12;
+use PublicInbox::TestCommon;
+use URI;
+require_mods 'IO::Socket::Socks';
+use_ok 'PublicInbox::NetNNTPSocks';
+my $url = $ENV{TEST_NNTP_ONION_URL} //
+	'nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta';
+my $uri = URI->new($url);
+my $on = PublicInbox::NetNNTPSocks->new_socks(
+	Port => $uri->port,
+	Host => $uri->host,
+	ProxyAddr => '127.0.0.1', # default Tor address + port
+	ProxyPort => 9050,
+) or xbail('err = '.eval('$IO::Socket::Socks::SOCKS_ERROR'));
+my ($nr, $min, $max, $grp) = $on->group($uri->group);
+ok($nr > 0 && $min > 0 && $min < $max, 'nr, min, max make sense') or
+	diag explain([$nr, $min, $max, $grp]);
+is($grp, $uri->group, 'group matches');
+done_testing;

^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [PATCH 8/8] net_reader: support (imap|nntp).proxy in config file
  2021-04-30  9:24 [PATCH 0/8] lei NNTP/IMAP .onion support and misc fixes Eric Wong
                   ` (6 preceding siblings ...)
  2021-04-30  9:24 ` [PATCH 7/8] net_reader: Net::NNTP --proxy=socks5h:// support Eric Wong
@ 2021-04-30  9:24 ` Eric Wong
  7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-04-30  9:24 UTC (permalink / raw)
  To: meta

This allows us to use URL-matching config in git and specify
proxies on a per-host basis.  git 2.26+ users may use wildcards
to enable Tor (on 127.0.0.1:9050) for all NNTP and IMAP .onion
domains.

My ~/.config/lei/config file has the following:

	[imap "imap://*.onion"]
		proxy = socks5h://127.0.0.1:9050
	[nntp "nntp://*.onion"]
		proxy = socks5h://127.0.0.1:9050
---
 lib/PublicInbox/NetReader.pm | 85 ++++++++++++++++++++----------------
 1 file changed, 48 insertions(+), 37 deletions(-)

diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index b2c4fee2..64910fe1 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -25,6 +25,35 @@ sub uri_section ($) {
 	$uri->scheme . '://' . $uri->authority;
 }
 
+sub socks_args ($) {
+	my ($val) = @_;
+	return if ($val // '') eq '';
+	if ($val =~ m!\Asocks5h:// (?: \[ ([^\]]+) \] | ([^:/]+) )
+					(?::([0-9]+))?/*\z!ix) {
+		my ($h, $p) = ($1 // $2, $3 + 0);
+		$h = '127.0.0.1' if $h eq '0';
+		eval { require IO::Socket::Socks } or die <<EOM;
+IO::Socket::Socks missing for socks5h://$h:$p
+EOM
+		return { ProxyAddr => $h, ProxyPort => $p };
+	}
+	die "$val not understood (only socks5h:// is supported)\n";
+}
+
+sub mic_new ($$$$) {
+	my ($self, $mic_arg, $sec, $uri) = @_;
+	my %socks;
+	my $sa = $self->{imap_opt}->{$sec}->{-proxy_cfg} || $self->{-proxy_cli};
+	if ($sa) {
+		my %opt = %$sa;
+		$opt{ConnectAddr} = delete $mic_arg->{Server};
+		$opt{ConnectPort} = delete $mic_arg->{Port};
+		$socks{Socket} = IO::Socket::Socks->new(%opt) or die
+			"E: <$$uri> ".eval('$IO::Socket::Socks::SOCKS_ERROR');
+	}
+	PublicInbox::IMAPClient->new(%$mic_arg, %socks);
+}
+
 sub auth_anon_cb { '' }; # for Mail::IMAPClient::Authcallback
 
 # mic_for may prompt the user and store auth info, prepares mic_get
@@ -40,7 +69,8 @@ sub mic_for ($$$$) { # mic = Mail::IMAPClient
 		username => $uri->user,
 		password => $uri->password,
 	}, 'PublicInbox::GitCredential';
-	my $common = $mic_args->{uri_section($uri)} // {};
+	my $sec = uri_section($uri);
+	my $common = $mic_args->{$sec} // {};
 	# IMAPClient and Net::Netrc both mishandles `0', so we pass `127.0.0.1'
 	my $host = $cred->{host};
 	$host = '127.0.0.1' if $host eq '0';
@@ -52,18 +82,8 @@ sub mic_for ($$$$) { # mic = Mail::IMAPClient
 		%$common, # may set Starttls, Compress, Debug ....
 	};
 	require PublicInbox::IMAPClient;
-	my %socks;
-	if ($lei && $lei->{socks5h}) {
-		my %opt = %{$lei->{socks5h}};
-		$opt{ConnectAddr} = delete $mic_arg->{Server};
-		$opt{ConnectPort} = delete $mic_arg->{Port};
-		$socks{Socket} = IO::Socket::Socks->new(%opt) or die
-			"E: <$url> ".eval('$IO::Socket::Socks::SOCKS_ERROR');
-		$self->{mic_socks5h} = \%opt;
-	}
-	my $mic = PublicInbox::IMAPClient->new(%$mic_arg, %socks) or
-		die "E: <$url> new: $@\n";
-
+	my $mic = mic_new($self, $mic_arg, $sec, $uri) or
+			die "E: <$url> new: $@\n";
 	# default to using STARTTLS if it's available, but allow
 	# it to be disabled since I usually connect to localhost
 	if (!$mic_arg->{Ssl} && !defined($mic_arg->{Starttls}) &&
@@ -90,7 +110,7 @@ sub mic_for ($$$$) { # mic = Mail::IMAPClient
 	my $err;
 	if ($mic->login && $mic->IsAuthenticated) {
 		# success! keep IMAPClient->new arg in case we get disconnected
-		$self->{mic_arg}->{uri_section($uri)} = $mic_arg;
+		$self->{mic_arg}->{$sec} = $mic_arg;
 	} else {
 		$err = "E: <$url> LOGIN: $@\n";
 		if ($cred && defined($cred->{password})) {
@@ -118,6 +138,7 @@ sub nn_new ($$$) {
 	my ($nn_arg, $nntp_opt, $uri) = @_;
 	my $nn;
 	if (defined $nn_arg->{ProxyAddr}) {
+		require PublicInbox::NetNNTPSocks;
 		eval { $nn = PublicInbox::NetNNTPSocks->new_socks(%$nn_arg) };
 		die "E: <$uri> $@\n" if $@;
 	} else {
@@ -176,10 +197,8 @@ sub nn_for ($$$$) { # nn = Net::NNTP
 		SSL => $uri->secure, # snews == nntps
 		%$common, # may Debug ....
 	};
-	if ($lei && $lei->{socks5h}) {
-		require PublicInbox::NetNNTPSocks;
-		%$nn_arg = (%$nn_arg, %{$lei->{socks5h}});
-	}
+	my $sa = $self->{-proxy_cli};
+	%$nn_arg = (%$nn_arg, %$sa) if $sa;
 	my $nn = nn_new($nn_arg, $nntp_opt, $uri);
 	if ($cred) {
 		$cred->fill($lei); # may prompt user here
@@ -268,6 +287,8 @@ sub imap_common_init ($;$) {
 		}
 		my $to = cfg_intvl($cfg, 'imap.timeout', $$uri);
 		$mic_args->{$sec}->{Timeout} = $to if $to;
+		my $sa = socks_args($cfg->urlmatch('imap.Proxy', $$uri));
+		$self->{imap_opt}->{$sec}->{-proxy_cfg} = $sa if $sa;
 		for my $k (qw(pollInterval idleInterval)) {
 			$to = cfg_intvl($cfg, "imap.$k", $$uri) // next;
 			$self->{imap_opt}->{$sec}->{$k} = $to;
@@ -309,12 +330,15 @@ sub nntp_common_init ($;$) {
 	my $nn_args = {}; # scheme://authority => Net::NNTP->new arg
 	for my $uri (@{$self->{nntp_order}}) {
 		my $sec = uri_section($uri);
+		my $args = $nn_args->{$sec} //= {};
 
 		# Debug and Timeout are passed to Net::NNTP->new
 		my $v = cfg_bool($cfg, 'nntp.Debug', $$uri);
-		$nn_args->{$sec}->{Debug} = $v if defined $v;
+		$args->{Debug} = $v if defined $v;
 		my $to = cfg_intvl($cfg, 'nntp.Timeout', $$uri);
-		$nn_args->{$sec}->{Timeout} = $to if $to;
+		$args->{Timeout} = $to if $to;
+		my $sa = socks_args($cfg->urlmatch('nntp.Proxy', $$uri));
+		%$args = (%$args, %$sa) if $sa;
 
 		# Net::NNTP post-connect commands
 		for my $k (qw(starttls compress)) {
@@ -322,7 +346,7 @@ sub nntp_common_init ($;$) {
 			$self->{nntp_opt}->{$sec}->{$k} = $v;
 		}
 
-		# internal option
+		# -watch internal option
 		for my $k (qw(pollInterval)) {
 			$to = cfg_intvl($cfg, "nntp.$k", $$uri) // next;
 			$self->{nntp_opt}->{$sec}->{$k} = $to;
@@ -363,16 +387,8 @@ sub errors {
 		eval { require Net::NNTP } or
 			die "Net::NNTP is required for NNTP:\n$@\n";
 	}
-	if ($lei && (($lei->{opt}->{proxy}//'') =~ m!\Asocks5h://
-				(?: \[ ([^\]]+) \] | ([^:/]+) )
-				(?::([0-9]+))?/?(?:,|\z)!ix)) {
-		my ($h, $p) = ($1 // $2, $3 + 0);
-		$h = '127.0.0.1' if $h eq '0';
-		eval { require IO::Socket::Socks } or die <<EOM;
-IO::Socket::Socks missing for socks5h://$h:$p
-EOM
-		$lei->{socks5h} = { ProxyAddr => $h, ProxyPort => $p };
-	}
+	my $sa = socks_args($lei ? $lei->{opt}->{proxy} : undef);
+	$self->{-proxy_cli} = $sa if $sa;
 	undef;
 }
 
@@ -537,12 +553,7 @@ sub mic_get {
 			$mic_arg->{Authcallback} = $self->can($cb_name);
 		}
 	}
-	my %socks;
-	if (my $s5h = $self->{mic_socks5h}) {
-		$socks{Socket} = IO::Socket::Socks->new(%$s5h) or die
-			"E: <$$uri> ".eval('$IO::Socket::Socks::SOCKS_ERROR');
-	}
-	my $mic = PublicInbox::IMAPClient->new(%$mic_arg, %socks);
+	my $mic = mic_new($self, $mic_arg, $sec, $uri);
 	$cached //= {}; # invalid placeholder if no cache enabled
 	$mic && $mic->IsConnected ? ($cached->{$sec} = $mic) : undef;
 }

^ permalink raw reply related	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2021-04-30  9:24 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-30  9:24 [PATCH 0/8] lei NNTP/IMAP .onion support and misc fixes Eric Wong
2021-04-30  9:24 ` [PATCH 1/8] lei sucks: preserve utsname.machine, add "x86" where appropriate Eric Wong
2021-04-30  9:24 ` [PATCH 2/8] lei_curl: improve correctness of LD_PRELOAD check Eric Wong
2021-04-30  9:24 ` [PATCH 3/8] lei: kill old PIDs when dropping Eric Wong
2021-04-30  9:24 ` [PATCH 4/8] lei: ensure autoflush(1) is on STDERR Eric Wong
2021-04-30  9:24 ` [PATCH 5/8] net_reader: {nn,mic}_for: use prototypes for internal subs Eric Wong
2021-04-30  9:24 ` [PATCH 6/8] lei: IMAP .onion support via --proxy=s switch Eric Wong
2021-04-30  9:24 ` [PATCH 7/8] net_reader: Net::NNTP --proxy=socks5h:// support Eric Wong
2021-04-30  9:24 ` [PATCH 8/8] net_reader: support (imap|nntp).proxy in config file Eric Wong

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).