* [PATCH 0/8] lei NNTP/IMAP .onion support and misc fixes
@ 2021-04-30 9:24 Eric Wong
2021-04-30 9:24 ` [PATCH 1/8] lei sucks: preserve utsname.machine, add "x86" where appropriate Eric Wong
` (7 more replies)
0 siblings, 8 replies; 9+ messages in thread
From: Eric Wong @ 2021-04-30 9:24 UTC (permalink / raw)
To: meta
Attempting to use torsocks(1) for NNTP or IMAP could get tricky.
Fortunately, IO::Socket::Socks is packaged for on CentOS 7,
FreeBSD, and Debian, so it seems to be a reasonable way to
support NNTP and IMAP Tor onions.
--proxy= (shared with curl) is supported for one-off
command-line use, but imap.proxy and nntp.proxy are both
supported along with URL-matching variants with git 1.8.5 (or
git 2.26 for wildcard URL matching).
Only socks5h:// proxies are supported (the default with
IO::Socket::Socks), which is what Tor uses. I doubt its worth
the effort (and potential for DNS request leaks) to support
prior versions of SOCKS in 2021.
Eric Wong (8):
lei sucks: preserve utsname.machine, add "x86" where appropriate
lei_curl: improve correctness of LD_PRELOAD check
lei: kill old PIDs when dropping
lei: ensure autoflush(1) is on STDERR
net_reader: {nn,mic}_for: use prototypes for internal subs
lei: IMAP .onion support via --proxy=s switch
net_reader: Net::NNTP --proxy=socks5h:// support
net_reader: support (imap|nntp).proxy in config file
MANIFEST | 2 +
lib/PublicInbox/Config.pm | 1 +
lib/PublicInbox/LEI.pm | 24 ++++++++---
lib/PublicInbox/LeiCurl.pm | 2 +-
lib/PublicInbox/LeiInput.pm | 2 +-
lib/PublicInbox/LeiSucks.pm | 3 +-
lib/PublicInbox/LeiToMail.pm | 4 +-
lib/PublicInbox/NetNNTPSocks.pm | 33 +++++++++++++++
lib/PublicInbox/NetReader.pm | 72 +++++++++++++++++++++++++++------
xt/net_nntp_socks.t | 22 ++++++++++
10 files changed, 141 insertions(+), 24 deletions(-)
create mode 100644 lib/PublicInbox/NetNNTPSocks.pm
create mode 100644 xt/net_nntp_socks.t
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 1/8] lei sucks: preserve utsname.machine, add "x86" where appropriate
2021-04-30 9:24 [PATCH 0/8] lei NNTP/IMAP .onion support and misc fixes Eric Wong
@ 2021-04-30 9:24 ` Eric Wong
2021-04-30 9:24 ` [PATCH 2/8] lei_curl: improve correctness of LD_PRELOAD check Eric Wong
` (6 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-04-30 9:24 UTC (permalink / raw)
To: meta
It's helpful for us to distinguish x86 kernels from x86_64
kernels when using an x86 userspace. OSes are dropping i386
support and only support i486 and newer, so "x86" is a more
appropriate description for that platform than "i386".
---
lib/PublicInbox/LeiSucks.pm | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiSucks.pm b/lib/PublicInbox/LeiSucks.pm
index d364a856..2ce64d62 100644
--- a/lib/PublicInbox/LeiSucks.pm
+++ b/lib/PublicInbox/LeiSucks.pm
@@ -18,7 +18,8 @@ sub lei_sucks {
$lei->start_pager if -t $lei->{1};
my ($os, undef, $rel, undef, $mac)= POSIX::uname();
if ($mac eq 'x86_64' && $Config{ptrsize} == 4) {
- $mac = $Config{cppsymbols} =~ /\b__ILP32__=1\b/ ? 'x32' : 'i386'
+ $mac .= $Config{cppsymbols} =~ /\b__ILP32__=1\b/ ?
+ ',u=x32' : ',u=x86';
}
eval { require PublicInbox };
my $pi_ver = eval('$PublicInbox::VERSION') // '(???)';
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 2/8] lei_curl: improve correctness of LD_PRELOAD check
2021-04-30 9:24 [PATCH 0/8] lei NNTP/IMAP .onion support and misc fixes Eric Wong
2021-04-30 9:24 ` [PATCH 1/8] lei sucks: preserve utsname.machine, add "x86" where appropriate Eric Wong
@ 2021-04-30 9:24 ` Eric Wong
2021-04-30 9:24 ` [PATCH 3/8] lei: kill old PIDs when dropping Eric Wong
` (5 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-04-30 9:24 UTC (permalink / raw)
To: meta
LD_PRELOAD sent by a client can't affect lei-daemon.
---
lib/PublicInbox/Config.pm | 1 +
lib/PublicInbox/LeiCurl.pm | 2 +-
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/Config.pm b/lib/PublicInbox/Config.pm
index 016f50ec..3f0f5a01 100644
--- a/lib/PublicInbox/Config.pm
+++ b/lib/PublicInbox/Config.pm
@@ -12,6 +12,7 @@ use strict;
use v5.10.1;
use PublicInbox::Inbox;
use PublicInbox::Spawn qw(popen_rd);
+our $LD_PRELOAD = $ENV{LD_PRELOAD}; # only valid at startup
sub _array ($) { ref($_[0]) eq 'ARRAY' ? $_[0] : [ $_[0] ] }
diff --git a/lib/PublicInbox/LeiCurl.pm b/lib/PublicInbox/LeiCurl.pm
index 69c64cdf..ce57e796 100644
--- a/lib/PublicInbox/LeiCurl.pm
+++ b/lib/PublicInbox/LeiCurl.pm
@@ -55,7 +55,7 @@ sub torsocks { # useful for "git clone" and "git fetch", too
$opt->{torsocks} = 'false' if $opt->{'no-torsocks'};
my $torsocks = $opt->{torsocks} //= 'auto';
if ($torsocks eq 'auto' && substr($uri->host, -6) eq '.onion' &&
- (($lei->{env}->{LD_PRELOAD}//'') !~ /torsocks/)) {
+ ($PublicInbox::Config::LD_PRELOAD//'') !~ m!/libtorsocks\b!) {
# "auto" continues anyways if torsocks is missing;
# a proxy may be specified via CLI, curlrc,
# environment variable, or even firewall rule
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 3/8] lei: kill old PIDs when dropping
2021-04-30 9:24 [PATCH 0/8] lei NNTP/IMAP .onion support and misc fixes Eric Wong
2021-04-30 9:24 ` [PATCH 1/8] lei sucks: preserve utsname.machine, add "x86" where appropriate Eric Wong
2021-04-30 9:24 ` [PATCH 2/8] lei_curl: improve correctness of LD_PRELOAD check Eric Wong
@ 2021-04-30 9:24 ` Eric Wong
2021-04-30 9:24 ` [PATCH 4/8] lei: ensure autoflush(1) is on STDERR Eric Wong
` (4 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-04-30 9:24 UTC (permalink / raw)
To: meta
This ensures hitting Ctrl-C on a long-running "lei convert" or
similar will stop the WQ worker, even after we've closed
the WQ socketpair in the daemon.
---
lib/PublicInbox/LEI.pm | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 52ce8ec2..3468094f 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -387,7 +387,14 @@ my @WQ_KEYS = qw(lxs l2m wq1); # internal workers
sub _drop_wq {
my ($self) = @_;
- for my $wq (grep(defined, delete(@$self{@WQ_KEYS}))) { $wq->DESTROY }
+ for my $wq (grep(defined, delete(@$self{@WQ_KEYS}))) {
+ if ($wq->wq_kill) {
+ $wq->wq_close(0, undef, $self);
+ } elsif ($wq->wq_kill_old) {
+ $wq->wq_wait_old(undef, $self);
+ }
+ $wq->DESTROY;
+ }
}
# pronounced "exit": x_it(1 << 8) => exit(1); x_it(13) => SIGPIPE
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 4/8] lei: ensure autoflush(1) is on STDERR
2021-04-30 9:24 [PATCH 0/8] lei NNTP/IMAP .onion support and misc fixes Eric Wong
` (2 preceding siblings ...)
2021-04-30 9:24 ` [PATCH 3/8] lei: kill old PIDs when dropping Eric Wong
@ 2021-04-30 9:24 ` Eric Wong
2021-04-30 9:24 ` [PATCH 5/8] net_reader: {nn,mic}_for: use prototypes for internal subs Eric Wong
` (3 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-04-30 9:24 UTC (permalink / raw)
To: meta
This fixes error reporting for oneshot tests in xt/lei-auth-failure.t
---
lib/PublicInbox/LEI.pm | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 3468094f..6a82d497 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -500,6 +500,7 @@ sub _lei_atfork_child {
}
} else { # worker, Net::NNTP (Net::Cmd) uses STDERR directly
open STDERR, '+>&='.fileno($self->{2}) or warn "open $!";
+ STDERR->autoflush(1);
}
close($_) for (grep(defined, delete @$self{qw(3 old_1 au_done)}));
if (my $op_c = delete $self->{pkt_op_c}) {
@@ -676,6 +677,7 @@ sub lazy_cb ($$$) {
sub dispatch {
my ($self, $cmd, @argv) = @_;
local $current_lei = $self; # for __WARN__
+ $self->{2}->autoflush(1); # keep stdout buffered until x_it|DESTROY
dump_and_clear_log("from previous run\n");
return _help($self, 'no command given') unless defined($cmd);
# do not support Getopt bundling for this
@@ -1006,7 +1008,6 @@ sub accept_dispatch { # Listener {post_accept} callback
}
$i == 4 or return send($sock, 'not enough FDs='.($i-1), MSG_EOR)
}
- $self->{2}->autoflush(1); # keep stdout buffered until x_it|DESTROY
# $ENV_STR = join('', map { "\0$_=$ENV{$_}" } keys %ENV);
# $buf = "$argc\0".join("\0", @ARGV).$ENV_STR."\0\0";
substr($buf, -2, 2, '') eq "\0\0" or # s/\0\0\z//
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 5/8] net_reader: {nn,mic}_for: use prototypes for internal subs
2021-04-30 9:24 [PATCH 0/8] lei NNTP/IMAP .onion support and misc fixes Eric Wong
` (3 preceding siblings ...)
2021-04-30 9:24 ` [PATCH 4/8] lei: ensure autoflush(1) is on STDERR Eric Wong
@ 2021-04-30 9:24 ` Eric Wong
2021-04-30 9:24 ` [PATCH 6/8] lei: IMAP .onion support via --proxy=s switch Eric Wong
` (2 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-04-30 9:24 UTC (permalink / raw)
To: meta
We don't use these subs elsewhere, so stick prototypes on them
to give them a little extra checking.
---
lib/PublicInbox/NetReader.pm | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index 3fc37b10..b9365c05 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -27,7 +27,7 @@ sub uri_section ($) {
sub auth_anon_cb { '' }; # for Mail::IMAPClient::Authcallback
# mic_for may prompt the user and store auth info, prepares mic_get
-sub mic_for { # mic = Mail::IMAPClient
+sub mic_for ($$$$) { # mic = Mail::IMAPClient
my ($self, $url, $mic_args, $lei) = @_;
require PublicInbox::URIimap;
my $uri = PublicInbox::URIimap->new($url);
@@ -133,7 +133,7 @@ E: <$uri> STARTTLS requested and failed
$nn;
}
-sub nn_for ($$$;$) { # nn = Net::NNTP
+sub nn_for ($$$$) { # nn = Net::NNTP
my ($self, $uri, $nn_args, $lei) = @_;
my $sec = uri_section($uri);
my $nntp_opt = $self->{nntp_opt}->{$sec} //= {};
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 6/8] lei: IMAP .onion support via --proxy=s switch
2021-04-30 9:24 [PATCH 0/8] lei NNTP/IMAP .onion support and misc fixes Eric Wong
` (4 preceding siblings ...)
2021-04-30 9:24 ` [PATCH 5/8] net_reader: {nn,mic}_for: use prototypes for internal subs Eric Wong
@ 2021-04-30 9:24 ` Eric Wong
2021-04-30 9:24 ` [PATCH 7/8] net_reader: Net::NNTP --proxy=socks5h:// support Eric Wong
2021-04-30 9:24 ` [PATCH 8/8] net_reader: support (imap|nntp).proxy in config file Eric Wong
7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-04-30 9:24 UTC (permalink / raw)
To: meta
Mail::IMAPClient provides the ability to pass a pre-connected
Socket to it. We can rely on this functionality to use
IO::Socket::Socks in place whatever socket class
Mail::IMAPClient chooses to use.
The --proxy=s is shared with curl(1), though we only support
socks5h:// at the moment. Is there any need for SOCKS4 or SOCKS5
without name resolution? Tor .onions require socks5h:// for
name resolution and to prevent data leakage.
---
lib/PublicInbox/LEI.pm | 12 ++++++++----
lib/PublicInbox/LeiInput.pm | 2 +-
lib/PublicInbox/LeiToMail.pm | 4 ++--
lib/PublicInbox/NetReader.pm | 31 ++++++++++++++++++++++++++++---
4 files changed, 39 insertions(+), 10 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 6a82d497..bb67fc0b 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -188,7 +188,8 @@ our %CMD = ( # sorted in order of importance/use:
qw(stdin| threads|t from|f=s mid=s oid=s), @c_opt ],
'tag' => [ 'KEYWORDS...',
'set/unset keywords and/or labels on message(s)',
- qw(stdin| in-format|F=s input|i=s@ oid=s@ mid=s@), @c_opt,
+ qw(stdin| in-format|F=s input|i=s@ oid=s@ mid=s@),
+ qw(no-torsocks torsocks=s), PublicInbox::LeiQuery::curl_opt(), @c_opt,
pass_through('-kw:foo for delete') ],
'forget' => [ '[--stdin|--oid=OID|--by-mid=MID]',
"exclude message(s) on stdin from `q' search results",
@@ -211,11 +212,12 @@ our %CMD = ( # sorted in order of importance/use:
'import' => [ 'LOCATION...|--stdin',
'one-time import/update from URL or filesystem',
qw(stdin| offset=i recursive|r exclude=s include|I=s
- lock=s@ in-format|F=s kw! verbose|v+ incremental! mail-sync!), @c_opt ],
+ lock=s@ in-format|F=s kw! verbose|v+ incremental! mail-sync!),
+ qw(no-torsocks torsocks=s), PublicInbox::LeiQuery::curl_opt(), @c_opt ],
'convert' => [ 'LOCATION...|--stdin',
'one-time conversion from URL or filesystem to another format',
- qw(stdin| in-format|F=s out-format|f=s output|mfolder|o=s
- lock=s@ kw!), @c_opt ],
+ qw(stdin| in-format|F=s out-format|f=s output|mfolder|o=s lock=s@ kw!),
+ qw(no-torsocks torsocks=s), PublicInbox::LeiQuery::curl_opt(), @c_opt ],
'p2q' => [ 'FILE|COMMIT_OID|--stdin',
"use a patch to generate a query for `lei q --stdin'",
qw(stdin| want|w=s@ uri debug), @c_opt ],
@@ -277,6 +279,8 @@ my %OPTDESC = (
'path-a|a=s' => 'pre-image pathname associated with OID',
'path-b|b=s' => 'post-image pathname associated with OID',
'git-dir=s@' => 'additional git repository to scan',
+'proxy=s' => [ 'PROTO://HOST[:PORT]', # shared with curl(1)
+ "proxy for (e.g. `socks5h://0:9050')" ],
'torsocks=s' => ['VAL|auto|no|yes',
'whether or not to wrap git and curl commands with torsocks'],
'no-torsocks' => 'alias for --torsocks=no',
diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm
index 277ad88d..86f300c3 100644
--- a/lib/PublicInbox/LeiInput.pm
+++ b/lib/PublicInbox/LeiInput.pm
@@ -294,7 +294,7 @@ $input is `eml', not --in-format=$in_fmt
}
if ($net) {
$net->{-can_die} = 1;
- if (my $err = $net->errors) {
+ if (my $err = $net->errors($lei)) {
return $lei->fail($err);
}
$net->{quiet} = $lei->{opt}->{quiet};
diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
index fa3af710..eda4701c 100644
--- a/lib/PublicInbox/LeiToMail.pm
+++ b/lib/PublicInbox/LeiToMail.pm
@@ -351,14 +351,14 @@ sub new {
require PublicInbox::MboxReader;
$self->can("eml2$fmt") or die "bad mbox format: $fmt\n";
$self->{base_type} = 'mbox';
- } elsif ($fmt =~ /\Aimaps?\z/) { # TODO .onion support
+ } elsif ($fmt =~ /\Aimaps?\z/) {
require PublicInbox::NetWriter;
require PublicInbox::URIimap;
my $net = PublicInbox::NetWriter->new;
$net->{quiet} = $lei->{opt}->{quiet};
my $uri = PublicInbox::URIimap->new($dst)->canonical;
$net->add_url($uri);
- my $err = $net->errors;
+ my $err = $net->errors($lei);
return $lei->fail($err) if $err;
$uri->mailbox or return $lei->fail("No mailbox: $dst");
$self->{uri} = $uri;
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index b9365c05..ac23e701 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -7,6 +7,7 @@ use strict;
use v5.10.1;
use parent qw(Exporter PublicInbox::IPC);
use PublicInbox::Eml;
+use PublicInbox::Config;
our %IMAPflags2kw = map {; "\\\u$_" => $_ } qw(seen answered flagged draft);
$IMAPflags2kw{'$Forwarded'} = 'forwarded'; # RFC 5550
@@ -51,7 +52,16 @@ sub mic_for ($$$$) { # mic = Mail::IMAPClient
%$common, # may set Starttls, Compress, Debug ....
};
require PublicInbox::IMAPClient;
- my $mic = PublicInbox::IMAPClient->new(%$mic_arg) or
+ my %socks;
+ if ($lei && $lei->{socks5h}) {
+ my %opt = %{$lei->{socks5h}};
+ $opt{ConnectAddr} = delete $mic_arg->{Server};
+ $opt{ConnectPort} = delete $mic_arg->{Port};
+ $socks{Socket} = IO::Socket::Socks->new(%opt) or die
+ "E: <$url> ".eval('$IO::Socket::Socks::SOCKS_ERROR');
+ $self->{mic_socks5h} = \%opt;
+ }
+ my $mic = PublicInbox::IMAPClient->new(%$mic_arg, %socks) or
die "E: <$url> new: $@\n";
# default to using STARTTLS if it's available, but allow
@@ -331,7 +341,7 @@ sub add_url {
}
sub errors {
- my ($self) = @_;
+ my ($self, $lei) = @_;
if (my $u = $self->{unsupported_url}) {
return "Unsupported URL(s): @$u";
}
@@ -343,6 +353,16 @@ sub errors {
eval { require Net::NNTP } or
die "Net::NNTP is required for NNTP:\n$@\n";
}
+ if ($lei && (($lei->{opt}->{proxy}//'') =~ m!\Asocks5h://
+ (?: \[ ([^\]]+) \] | ([^:/]+) )
+ (?::([0-9]+))?/?(?:,|\z)!ix)) {
+ my ($h, $p) = ($1 // $2, $3 + 0);
+ $h = '127.0.0.1' if $h eq '0';
+ eval { require IO::Socket::Socks } or die <<EOM;
+IO::Socket::Socks missing for socks5h://$h:$p
+EOM
+ $lei->{socks5h} = { ProxyAddr => $h, ProxyPort => $p };
+ }
undef;
}
@@ -507,7 +527,12 @@ sub mic_get {
$mic_arg->{Authcallback} = $self->can($cb_name);
}
}
- my $mic = PublicInbox::IMAPClient->new(%$mic_arg);
+ my %socks;
+ if (my $s5h = $self->{mic_socks5h}) {
+ $socks{Socket} = IO::Socket::Socks->new(%$s5h) or die
+ "E: <$$uri> ".eval('$IO::Socket::Socks::SOCKS_ERROR');
+ }
+ my $mic = PublicInbox::IMAPClient->new(%$mic_arg, %socks);
$cached //= {}; # invalid placeholder if no cache enabled
$mic && $mic->IsConnected ? ($cached->{$sec} = $mic) : undef;
}
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 7/8] net_reader: Net::NNTP --proxy=socks5h:// support
2021-04-30 9:24 [PATCH 0/8] lei NNTP/IMAP .onion support and misc fixes Eric Wong
` (5 preceding siblings ...)
2021-04-30 9:24 ` [PATCH 6/8] lei: IMAP .onion support via --proxy=s switch Eric Wong
@ 2021-04-30 9:24 ` Eric Wong
2021-04-30 9:24 ` [PATCH 8/8] net_reader: support (imap|nntp).proxy in config file Eric Wong
7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-04-30 9:24 UTC (permalink / raw)
To: meta
Since Net::NNTP doesn't support Socket or RawSocket
options/accessors like Mail::IMAPClient does; we must perform
localized @ISA manipulation and massage Net::NNTP into using
IO::Socket::Socks rather than IO::Socket::IP.
This is a bit fragile, but Net::Cmd and Net::NNTP rarely change;
and I keep an eye on them, anyways.
---
MANIFEST | 2 ++
lib/PublicInbox/NetNNTPSocks.pm | 33 +++++++++++++++++++++++++++++++++
lib/PublicInbox/NetReader.pm | 12 +++++++++++-
xt/net_nntp_socks.t | 22 ++++++++++++++++++++++
4 files changed, 68 insertions(+), 1 deletion(-)
create mode 100644 lib/PublicInbox/NetNNTPSocks.pm
create mode 100644 xt/net_nntp_socks.t
diff --git a/MANIFEST b/MANIFEST
index 5933ddf4..bc2ad671 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -241,6 +241,7 @@ lib/PublicInbox/NDC_PP.pm
lib/PublicInbox/NNTP.pm
lib/PublicInbox/NNTPD.pm
lib/PublicInbox/NNTPdeflate.pm
+lib/PublicInbox/NetNNTPSocks.pm
lib/PublicInbox/NetReader.pm
lib/PublicInbox/NetWriter.pm
lib/PublicInbox/NewsWWW.pm
@@ -518,6 +519,7 @@ xt/lei-auth-fail.t
xt/mem-imapd-tls.t
xt/mem-msgview.t
xt/msgtime_cmp.t
+xt/net_nntp_socks.t
xt/net_writer-imap.t
xt/nntpd-validate.t
xt/perf-msgview.t
diff --git a/lib/PublicInbox/NetNNTPSocks.pm b/lib/PublicInbox/NetNNTPSocks.pm
new file mode 100644
index 00000000..8495204a
--- /dev/null
+++ b/lib/PublicInbox/NetNNTPSocks.pm
@@ -0,0 +1,33 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# wrap Net::NNTP client with SOCKS support
+package PublicInbox::NetNNTPSocks;
+use strict;
+use v5.10.1;
+use Net::NNTP;
+our %OPT;
+our @ISA = qw(IO::Socket::Socks);
+my @SOCKS_KEYS = qw(ProxyAddr ProxyPort SocksVersion SocksDebug SocksResolve);
+
+# use this instead of Net::NNTP->new if using Proxy*
+sub new_socks {
+ my (undef, %opt) = @_;
+ require IO::Socket::Socks;
+ local @Net::NNTP::ISA = (qw(Net::Cmd), __PACKAGE__);
+ local %OPT = map {;
+ defined($opt{$_}) ? ($_ => $opt{$_}) : ()
+ } @SOCKS_KEYS;
+ Net::NNTP->new(%opt); # this calls our new() below:
+}
+
+# called by Net::NNTP->new
+sub new {
+ my ($self, %opt) = @_;
+ @OPT{qw(ConnectAddr ConnectPort)} = @opt{qw(PeerAddr PeerPort)};
+ my $ret = $self->SUPER::new(%OPT) or
+ die 'SOCKS error: '.eval('$IO::Socket::Socks::SOCKS_ERROR');
+ $ret;
+}
+
+1;
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index ac23e701..b2c4fee2 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -116,7 +116,13 @@ sub try_starttls ($) {
sub nn_new ($$$) {
my ($nn_arg, $nntp_opt, $uri) = @_;
- my $nn = Net::NNTP->new(%$nn_arg) or die "E: <$uri> new: $!\n";
+ my $nn;
+ if (defined $nn_arg->{ProxyAddr}) {
+ eval { $nn = PublicInbox::NetNNTPSocks->new_socks(%$nn_arg) };
+ die "E: <$uri> $@\n" if $@;
+ } else {
+ $nn = Net::NNTP->new(%$nn_arg) or die "E: <$uri> new: $!\n";
+ }
# default to using STARTTLS if it's available, but allow
# it to be disabled for localhost/VPN users
@@ -170,6 +176,10 @@ sub nn_for ($$$$) { # nn = Net::NNTP
SSL => $uri->secure, # snews == nntps
%$common, # may Debug ....
};
+ if ($lei && $lei->{socks5h}) {
+ require PublicInbox::NetNNTPSocks;
+ %$nn_arg = (%$nn_arg, %{$lei->{socks5h}});
+ }
my $nn = nn_new($nn_arg, $nntp_opt, $uri);
if ($cred) {
$cred->fill($lei); # may prompt user here
diff --git a/xt/net_nntp_socks.t b/xt/net_nntp_socks.t
new file mode 100644
index 00000000..4a144fd8
--- /dev/null
+++ b/xt/net_nntp_socks.t
@@ -0,0 +1,22 @@
+#!perl -w
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+use v5.12;
+use PublicInbox::TestCommon;
+use URI;
+require_mods 'IO::Socket::Socks';
+use_ok 'PublicInbox::NetNNTPSocks';
+my $url = $ENV{TEST_NNTP_ONION_URL} //
+ 'nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta';
+my $uri = URI->new($url);
+my $on = PublicInbox::NetNNTPSocks->new_socks(
+ Port => $uri->port,
+ Host => $uri->host,
+ ProxyAddr => '127.0.0.1', # default Tor address + port
+ ProxyPort => 9050,
+) or xbail('err = '.eval('$IO::Socket::Socks::SOCKS_ERROR'));
+my ($nr, $min, $max, $grp) = $on->group($uri->group);
+ok($nr > 0 && $min > 0 && $min < $max, 'nr, min, max make sense') or
+ diag explain([$nr, $min, $max, $grp]);
+is($grp, $uri->group, 'group matches');
+done_testing;
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 8/8] net_reader: support (imap|nntp).proxy in config file
2021-04-30 9:24 [PATCH 0/8] lei NNTP/IMAP .onion support and misc fixes Eric Wong
` (6 preceding siblings ...)
2021-04-30 9:24 ` [PATCH 7/8] net_reader: Net::NNTP --proxy=socks5h:// support Eric Wong
@ 2021-04-30 9:24 ` Eric Wong
7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-04-30 9:24 UTC (permalink / raw)
To: meta
This allows us to use URL-matching config in git and specify
proxies on a per-host basis. git 2.26+ users may use wildcards
to enable Tor (on 127.0.0.1:9050) for all NNTP and IMAP .onion
domains.
My ~/.config/lei/config file has the following:
[imap "imap://*.onion"]
proxy = socks5h://127.0.0.1:9050
[nntp "nntp://*.onion"]
proxy = socks5h://127.0.0.1:9050
---
lib/PublicInbox/NetReader.pm | 85 ++++++++++++++++++++----------------
1 file changed, 48 insertions(+), 37 deletions(-)
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index b2c4fee2..64910fe1 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -25,6 +25,35 @@ sub uri_section ($) {
$uri->scheme . '://' . $uri->authority;
}
+sub socks_args ($) {
+ my ($val) = @_;
+ return if ($val // '') eq '';
+ if ($val =~ m!\Asocks5h:// (?: \[ ([^\]]+) \] | ([^:/]+) )
+ (?::([0-9]+))?/*\z!ix) {
+ my ($h, $p) = ($1 // $2, $3 + 0);
+ $h = '127.0.0.1' if $h eq '0';
+ eval { require IO::Socket::Socks } or die <<EOM;
+IO::Socket::Socks missing for socks5h://$h:$p
+EOM
+ return { ProxyAddr => $h, ProxyPort => $p };
+ }
+ die "$val not understood (only socks5h:// is supported)\n";
+}
+
+sub mic_new ($$$$) {
+ my ($self, $mic_arg, $sec, $uri) = @_;
+ my %socks;
+ my $sa = $self->{imap_opt}->{$sec}->{-proxy_cfg} || $self->{-proxy_cli};
+ if ($sa) {
+ my %opt = %$sa;
+ $opt{ConnectAddr} = delete $mic_arg->{Server};
+ $opt{ConnectPort} = delete $mic_arg->{Port};
+ $socks{Socket} = IO::Socket::Socks->new(%opt) or die
+ "E: <$$uri> ".eval('$IO::Socket::Socks::SOCKS_ERROR');
+ }
+ PublicInbox::IMAPClient->new(%$mic_arg, %socks);
+}
+
sub auth_anon_cb { '' }; # for Mail::IMAPClient::Authcallback
# mic_for may prompt the user and store auth info, prepares mic_get
@@ -40,7 +69,8 @@ sub mic_for ($$$$) { # mic = Mail::IMAPClient
username => $uri->user,
password => $uri->password,
}, 'PublicInbox::GitCredential';
- my $common = $mic_args->{uri_section($uri)} // {};
+ my $sec = uri_section($uri);
+ my $common = $mic_args->{$sec} // {};
# IMAPClient and Net::Netrc both mishandles `0', so we pass `127.0.0.1'
my $host = $cred->{host};
$host = '127.0.0.1' if $host eq '0';
@@ -52,18 +82,8 @@ sub mic_for ($$$$) { # mic = Mail::IMAPClient
%$common, # may set Starttls, Compress, Debug ....
};
require PublicInbox::IMAPClient;
- my %socks;
- if ($lei && $lei->{socks5h}) {
- my %opt = %{$lei->{socks5h}};
- $opt{ConnectAddr} = delete $mic_arg->{Server};
- $opt{ConnectPort} = delete $mic_arg->{Port};
- $socks{Socket} = IO::Socket::Socks->new(%opt) or die
- "E: <$url> ".eval('$IO::Socket::Socks::SOCKS_ERROR');
- $self->{mic_socks5h} = \%opt;
- }
- my $mic = PublicInbox::IMAPClient->new(%$mic_arg, %socks) or
- die "E: <$url> new: $@\n";
-
+ my $mic = mic_new($self, $mic_arg, $sec, $uri) or
+ die "E: <$url> new: $@\n";
# default to using STARTTLS if it's available, but allow
# it to be disabled since I usually connect to localhost
if (!$mic_arg->{Ssl} && !defined($mic_arg->{Starttls}) &&
@@ -90,7 +110,7 @@ sub mic_for ($$$$) { # mic = Mail::IMAPClient
my $err;
if ($mic->login && $mic->IsAuthenticated) {
# success! keep IMAPClient->new arg in case we get disconnected
- $self->{mic_arg}->{uri_section($uri)} = $mic_arg;
+ $self->{mic_arg}->{$sec} = $mic_arg;
} else {
$err = "E: <$url> LOGIN: $@\n";
if ($cred && defined($cred->{password})) {
@@ -118,6 +138,7 @@ sub nn_new ($$$) {
my ($nn_arg, $nntp_opt, $uri) = @_;
my $nn;
if (defined $nn_arg->{ProxyAddr}) {
+ require PublicInbox::NetNNTPSocks;
eval { $nn = PublicInbox::NetNNTPSocks->new_socks(%$nn_arg) };
die "E: <$uri> $@\n" if $@;
} else {
@@ -176,10 +197,8 @@ sub nn_for ($$$$) { # nn = Net::NNTP
SSL => $uri->secure, # snews == nntps
%$common, # may Debug ....
};
- if ($lei && $lei->{socks5h}) {
- require PublicInbox::NetNNTPSocks;
- %$nn_arg = (%$nn_arg, %{$lei->{socks5h}});
- }
+ my $sa = $self->{-proxy_cli};
+ %$nn_arg = (%$nn_arg, %$sa) if $sa;
my $nn = nn_new($nn_arg, $nntp_opt, $uri);
if ($cred) {
$cred->fill($lei); # may prompt user here
@@ -268,6 +287,8 @@ sub imap_common_init ($;$) {
}
my $to = cfg_intvl($cfg, 'imap.timeout', $$uri);
$mic_args->{$sec}->{Timeout} = $to if $to;
+ my $sa = socks_args($cfg->urlmatch('imap.Proxy', $$uri));
+ $self->{imap_opt}->{$sec}->{-proxy_cfg} = $sa if $sa;
for my $k (qw(pollInterval idleInterval)) {
$to = cfg_intvl($cfg, "imap.$k", $$uri) // next;
$self->{imap_opt}->{$sec}->{$k} = $to;
@@ -309,12 +330,15 @@ sub nntp_common_init ($;$) {
my $nn_args = {}; # scheme://authority => Net::NNTP->new arg
for my $uri (@{$self->{nntp_order}}) {
my $sec = uri_section($uri);
+ my $args = $nn_args->{$sec} //= {};
# Debug and Timeout are passed to Net::NNTP->new
my $v = cfg_bool($cfg, 'nntp.Debug', $$uri);
- $nn_args->{$sec}->{Debug} = $v if defined $v;
+ $args->{Debug} = $v if defined $v;
my $to = cfg_intvl($cfg, 'nntp.Timeout', $$uri);
- $nn_args->{$sec}->{Timeout} = $to if $to;
+ $args->{Timeout} = $to if $to;
+ my $sa = socks_args($cfg->urlmatch('nntp.Proxy', $$uri));
+ %$args = (%$args, %$sa) if $sa;
# Net::NNTP post-connect commands
for my $k (qw(starttls compress)) {
@@ -322,7 +346,7 @@ sub nntp_common_init ($;$) {
$self->{nntp_opt}->{$sec}->{$k} = $v;
}
- # internal option
+ # -watch internal option
for my $k (qw(pollInterval)) {
$to = cfg_intvl($cfg, "nntp.$k", $$uri) // next;
$self->{nntp_opt}->{$sec}->{$k} = $to;
@@ -363,16 +387,8 @@ sub errors {
eval { require Net::NNTP } or
die "Net::NNTP is required for NNTP:\n$@\n";
}
- if ($lei && (($lei->{opt}->{proxy}//'') =~ m!\Asocks5h://
- (?: \[ ([^\]]+) \] | ([^:/]+) )
- (?::([0-9]+))?/?(?:,|\z)!ix)) {
- my ($h, $p) = ($1 // $2, $3 + 0);
- $h = '127.0.0.1' if $h eq '0';
- eval { require IO::Socket::Socks } or die <<EOM;
-IO::Socket::Socks missing for socks5h://$h:$p
-EOM
- $lei->{socks5h} = { ProxyAddr => $h, ProxyPort => $p };
- }
+ my $sa = socks_args($lei ? $lei->{opt}->{proxy} : undef);
+ $self->{-proxy_cli} = $sa if $sa;
undef;
}
@@ -537,12 +553,7 @@ sub mic_get {
$mic_arg->{Authcallback} = $self->can($cb_name);
}
}
- my %socks;
- if (my $s5h = $self->{mic_socks5h}) {
- $socks{Socket} = IO::Socket::Socks->new(%$s5h) or die
- "E: <$$uri> ".eval('$IO::Socket::Socks::SOCKS_ERROR');
- }
- my $mic = PublicInbox::IMAPClient->new(%$mic_arg, %socks);
+ my $mic = mic_new($self, $mic_arg, $sec, $uri);
$cached //= {}; # invalid placeholder if no cache enabled
$mic && $mic->IsConnected ? ($cached->{$sec} = $mic) : undef;
}
^ permalink raw reply related [flat|nested] 9+ messages in thread
end of thread, other threads:[~2021-04-30 9:24 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-30 9:24 [PATCH 0/8] lei NNTP/IMAP .onion support and misc fixes Eric Wong
2021-04-30 9:24 ` [PATCH 1/8] lei sucks: preserve utsname.machine, add "x86" where appropriate Eric Wong
2021-04-30 9:24 ` [PATCH 2/8] lei_curl: improve correctness of LD_PRELOAD check Eric Wong
2021-04-30 9:24 ` [PATCH 3/8] lei: kill old PIDs when dropping Eric Wong
2021-04-30 9:24 ` [PATCH 4/8] lei: ensure autoflush(1) is on STDERR Eric Wong
2021-04-30 9:24 ` [PATCH 5/8] net_reader: {nn,mic}_for: use prototypes for internal subs Eric Wong
2021-04-30 9:24 ` [PATCH 6/8] lei: IMAP .onion support via --proxy=s switch Eric Wong
2021-04-30 9:24 ` [PATCH 7/8] net_reader: Net::NNTP --proxy=socks5h:// support Eric Wong
2021-04-30 9:24 ` [PATCH 8/8] net_reader: support (imap|nntp).proxy in config file Eric Wong
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).