unofficial mirror of meta@public-inbox.org
 help / color / mirror / Atom feed
* [PATCH 0/4] lei: some net-related things
@ 2021-09-10  9:08 Eric Wong
  2021-09-10  9:08 ` [PATCH 1/4] lei_query: fix comment about %lei2curl commands Eric Wong
                   ` (3 more replies)
  0 siblings, 4 replies; 5+ messages in thread
From: Eric Wong @ 2021-09-10  9:08 UTC (permalink / raw)
  To: meta

After some consideration, ~/.netrc will no longer be read by
default to match the behavior of existing IMAP/NNTP clients.

And lei-index is pretty limited, but still useful for Maildir
users, so it's documented (mainly for its limitations).

Eric Wong (4):
  lei_query: fix comment about %lei2curl commands
  lei: split out @net_opt for curl/torsocks use
  lei: do not read ~/.netrc by default
  doc: lei-index manpage

 Documentation/lei-index.pod      | 69 ++++++++++++++++++++++++++++++++
 MANIFEST                         |  1 +
 Makefile.PL                      |  2 +-
 lib/PublicInbox/GitCredential.pm |  8 +++-
 lib/PublicInbox/LEI.pm           | 34 ++++++++--------
 lib/PublicInbox/LeiQuery.pm      |  2 +-
 lib/PublicInbox/NetReader.pm     |  4 +-
 7 files changed, 96 insertions(+), 24 deletions(-)
 create mode 100644 Documentation/lei-index.pod

^ permalink raw reply	[flat|nested] 5+ messages in thread

* [PATCH 1/4] lei_query: fix comment about %lei2curl commands
  2021-09-10  9:08 [PATCH 0/4] lei: some net-related things Eric Wong
@ 2021-09-10  9:08 ` Eric Wong
  2021-09-10  9:08 ` [PATCH 2/4] lei: split out @net_opt for curl/torsocks use Eric Wong
                   ` (2 subsequent siblings)
  3 siblings, 0 replies; 5+ messages in thread
From: Eric Wong @ 2021-09-10  9:08 UTC (permalink / raw)
  To: meta

Just a typo.
---
 lib/PublicInbox/LeiQuery.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/PublicInbox/LeiQuery.pm b/lib/PublicInbox/LeiQuery.pm
index 962ad49e..d5f132f1 100644
--- a/lib/PublicInbox/LeiQuery.pm
+++ b/lib/PublicInbox/LeiQuery.pm
@@ -190,7 +190,7 @@ sub _complete_q {
 # analogues in git(1)? that would reduce likelyhood of conflicts with
 # our other CLI options
 # Note: some names are renamed to avoid potential conflicts,
-# see %lei2mail in lib/PublicInbox/LeiCurl.pm
+# see %lei2curl in lib/PublicInbox/LeiCurl.pm
 sub curl_opt { qw(
 	curl-config=s@
 	abstract-unix-socket=s anyauth basic cacert=s capath=s

^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [PATCH 2/4] lei: split out @net_opt for curl/torsocks use
  2021-09-10  9:08 [PATCH 0/4] lei: some net-related things Eric Wong
  2021-09-10  9:08 ` [PATCH 1/4] lei_query: fix comment about %lei2curl commands Eric Wong
@ 2021-09-10  9:08 ` Eric Wong
  2021-09-10  9:08 ` [PATCH 3/4] lei: do not read ~/.netrc by default Eric Wong
  2021-09-10  9:08 ` [PATCH 4/4] doc: lei-index manpage Eric Wong
  3 siblings, 0 replies; 5+ messages in thread
From: Eric Wong @ 2021-09-10  9:08 UTC (permalink / raw)
  To: meta

IMAP and NNTP connections share some curl(1) options for TLS,
IPv4/IPv6, or netrc, etc...
---
 lib/PublicInbox/LEI.pm | 34 ++++++++++++++++------------------
 1 file changed, 16 insertions(+), 18 deletions(-)

diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 3dce0236..bbb6ab7e 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -147,9 +147,9 @@ sub index_opt {
 }
 
 my @c_opt = qw(c=s@ C=s@ quiet|q);
-my @lxs_opt = (qw(remote! local! external! include|I=s@ exclude=s@ only=s@
-	import-remote! no-torsocks torsocks=s),
-	PublicInbox::LeiQuery::curl_opt());
+my @net_opt = (qw(no-torsocks torsocks=s), PublicInbox::LeiQuery::curl_opt());
+my @lxs_opt = qw(remote! local! external! include|I=s@ exclude=s@ only=s@
+	import-remote!);
 
 # we don't support -C as an alias for --find-copies since it's already
 # used for chdir
@@ -174,7 +174,7 @@ our @diff_opt = qw(unified|U=i output-indicator-new=s output-indicator-old=s
 our %CMD = ( # sorted in order of importance/use:
 'q' => [ '--stdin|SEARCH_TERMS...', 'search for messages matching terms',
 	'stdin|', # /|\z/ must be first for lone dash
-	@lxs_opt,
+	@lxs_opt, @net_opt,
 	qw(save! output|mfolder|o=s format|f=s dedupe|d=s threads|t+
 	sort|s=s reverse|r offset=i pretty jobs|j=s globoff|g augment|a
 	import-before! lock=s@ rsyncable alert=s@ mua=s verbose|v+
@@ -186,26 +186,26 @@ our %CMD = ( # sorted in order of importance/use:
 'lcat' => [ '--stdin|MSGID_OR_URL...', 'display local copy of message(s)',
 	'stdin|', # /|\z/ must be first for lone dash
 	# some of these options are ridiculous for lcat
-	@lxs_opt, qw(output|mfolder|o=s format|f=s dedupe|d=s threads|t+
+	@lxs_opt, @net_opt,
+	qw(output|mfolder|o=s format|f=s dedupe|d=s threads|t+
 	sort|s=s reverse|r offset=i jobs|j=s globoff|g augment|a
 	import-before! lock=s@ rsyncable alert=s@ mua=s verbose|v+
 	color!), @c_opt, opt_dash('limit|n=i', '[0-9]+') ],
 
 'blob' => [ 'OID', 'show a git blob, reconstructing from mail if necessary',
 	qw(git-dir=s@ cwd! verbose|v+ mail! oid-a|A=s path-a|a=s path-b|b=s),
-	@lxs_opt, @c_opt ],
+	@lxs_opt, @net_opt, @c_opt ],
 
 'rediff' => [ '--stdin|LOCATION...',
 		'regenerate a diff with different options',
 	'stdin|', # /|\z/ must be first for lone dash
 	qw(git-dir=s@ cwd! verbose|v+ color:s no-color),
-	@diff_opt, @lxs_opt, @c_opt ],
+	@diff_opt, @lxs_opt, @net_opt, @c_opt ],
 
 'add-external' => [ 'LOCATION',
 	'add/set priority of a publicinbox|extindex for extra matches',
-	qw(boost=i mirror=s no-torsocks torsocks=s inbox-version=i
-	verbose|v+), @c_opt, index_opt(),
-	PublicInbox::LeiQuery::curl_opt() ],
+	qw(boost=i mirror=s inbox-version=i verbose|v+),
+	@c_opt, index_opt(), @net_opt ],
 'ls-external' => [ '[FILTER]', 'list publicinbox|extindex locations',
 	qw(format|f=s z|0 globoff|g invert-match|v local remote), @c_opt ],
 'ls-label' => [ '', 'list labels', qw(z|0 stats:s), @c_opt ],
@@ -226,15 +226,14 @@ our %CMD = ( # sorted in order of importance/use:
 'rm' => [ '--stdin|LOCATION...',
 	'remove a message from the index and prevent reindexing',
 	'stdin|', # /|\z/ must be first for lone dash
-	qw(in-format|F=s lock=s@), @c_opt ],
+	qw(in-format|F=s lock=s@), @net_opt, @c_opt ],
 'plonk' => [ '--threads|--from=IDENT',
 	'exclude mail matching From: or threads from non-Message-ID searches',
 	qw(stdin| threads|t from|f=s mid=s oid=s), @c_opt ],
 'tag' => [ 'KEYWORDS...',
 	'set/unset keywords and/or labels on message(s)',
 	qw(stdin| in-format|F=s input|i=s@ oid=s@ mid=s@),
-	qw(no-torsocks torsocks=s), PublicInbox::LeiQuery::curl_opt(), @c_opt,
-	pass_through('-kw:foo for delete') ],
+	@net_opt, @c_opt, pass_through('-kw:foo for delete') ],
 
 'purge-mailsource' => [ 'LOCATION|--all',
 	'remove imported messages from IMAP, Maildirs, and MH',
@@ -253,25 +252,24 @@ our %CMD = ( # sorted in order of importance/use:
 
 'index' => [ 'LOCATION...', 'one-time index from URL or filesystem',
 	qw(in-format|F=s kw! offset=i recursive|r exclude=s include|I=s
-	verbose|v+ incremental!),
-	 PublicInbox::LeiQuery::curl_opt(), # mainly for --proxy=
+	verbose|v+ incremental!), @net_opt, # mainly for --proxy=
 	 @c_opt ],
 'import' => [ 'LOCATION...|--stdin',
 	'one-time import/update from URL or filesystem',
 	qw(stdin| offset=i recursive|r exclude=s include|I=s jobs=s new-only
 	lock=s@ in-format|F=s kw! verbose|v+ incremental! mail-sync!),
-	qw(no-torsocks torsocks=s), PublicInbox::LeiQuery::curl_opt(), @c_opt ],
+	@net_opt, @c_opt ],
 'forget-mail-sync' => [ 'LOCATION...',
 	'forget sync information for a mail folder', @c_opt ],
 'prune-mail-sync' => [ 'LOCATION...|--all',
 	'prune dangling sync data for a mail folder', 'all:s', @c_opt ],
 'export-kw' => [ 'LOCATION...|--all',
 	'one-time export of keywords of sync sources',
-	qw(all:s mode=s), @c_opt ],
+	qw(all:s mode=s), @net_opt, @c_opt ],
 'convert' => [ 'LOCATION...|--stdin',
 	'one-time conversion from URL or filesystem to another format',
 	qw(stdin| in-format|F=s out-format|f=s output|mfolder|o=s lock=s@ kw!),
-	qw(no-torsocks torsocks=s), PublicInbox::LeiQuery::curl_opt(), @c_opt ],
+	@net_opt, @c_opt ],
 'p2q' => [ 'FILE|COMMIT_OID|--stdin',
 	"use a patch to generate a query for `lei q --stdin'",
 	qw(stdin| want|w=s@ uri debug), @c_opt ],

^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [PATCH 3/4] lei: do not read ~/.netrc by default
  2021-09-10  9:08 [PATCH 0/4] lei: some net-related things Eric Wong
  2021-09-10  9:08 ` [PATCH 1/4] lei_query: fix comment about %lei2curl commands Eric Wong
  2021-09-10  9:08 ` [PATCH 2/4] lei: split out @net_opt for curl/torsocks use Eric Wong
@ 2021-09-10  9:08 ` Eric Wong
  2021-09-10  9:08 ` [PATCH 4/4] doc: lei-index manpage Eric Wong
  3 siblings, 0 replies; 5+ messages in thread
From: Eric Wong @ 2021-09-10  9:08 UTC (permalink / raw)
  To: meta

Since ~/.netrc isn't widely used by most (if any) NNTP and IMAP
clients, we won't read it by default for lei.  AFAIK, ~/.netrc
is mainly by FTP clients (e.g. ftp(1) and lftp(1)).  wget uses
it by default for HTTP(S) (and FTP), but curl does not.

To avoid breaking stable release use cases, public-inbox-watch
continues to read ~/.netrc by default.

The --netrc switch is supported by all existing lei commands
which may use curl.
---
 lib/PublicInbox/GitCredential.pm | 8 ++++++--
 lib/PublicInbox/NetReader.pm     | 4 ++--
 2 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/lib/PublicInbox/GitCredential.pm b/lib/PublicInbox/GitCredential.pm
index c83fed43..b18bba1e 100644
--- a/lib/PublicInbox/GitCredential.pm
+++ b/lib/PublicInbox/GitCredential.pm
@@ -31,8 +31,12 @@ sub run ($$;$) {
 	close $out_r or die "`git credential $op' failed: \$!=$! \$?=$?\n";
 }
 
-sub check_netrc ($) {
-	my ($self) = @_;
+sub check_netrc {
+	my ($self, $lei) = @_;
+
+	# n.b. lei doesn't load ~/.netrc by default, public-inbox-watch does,
+	# which may've been a mistake, but we have to live with it.
+	return if ($lei && !$lei->{opt}->{netrc});
 
 	# part of the standard library, but distributions may split it out
 	eval { require Net::Netrc };
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index a0e52fc5..f0f56431 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -96,7 +96,7 @@ sub mic_for ($$$$) { # mic = Mail::IMAPClient
 		$cred = undef;
 	}
 	if ($cred) {
-		my $p = $cred->{password} // $cred->check_netrc;
+		my $p = $cred->{password} // $cred->check_netrc($lei);
 		$cred->fill($lei) unless defined($p); # may prompt user here
 		$mic->User($mic_arg->{User} = $cred->{username});
 		$mic->Password($mic_arg->{Password} = $cred->{password});
@@ -191,7 +191,7 @@ sub nn_for ($$$$) { # nn = Net::NNTP
 		}, 'PublicInbox::GitCredential';
 		($u, $p) = split(/:/, $ui, 2);
 		($cred->{username}, $cred->{password}) = ($u, $p);
-		$p //= $cred->check_netrc;
+		$p //= $cred->check_netrc($lei);
 	}
 	my $common = $nn_common->{$sec} // {};
 	my $nn_arg = {

^ permalink raw reply related	[flat|nested] 5+ messages in thread

* [PATCH 4/4] doc: lei-index manpage
  2021-09-10  9:08 [PATCH 0/4] lei: some net-related things Eric Wong
                   ` (2 preceding siblings ...)
  2021-09-10  9:08 ` [PATCH 3/4] lei: do not read ~/.netrc by default Eric Wong
@ 2021-09-10  9:08 ` Eric Wong
  3 siblings, 0 replies; 5+ messages in thread
From: Eric Wong @ 2021-09-10  9:08 UTC (permalink / raw)
  To: meta

It's a pretty incomplete command, so it's important to document
its incompleteness.
---
 Documentation/lei-index.pod | 69 +++++++++++++++++++++++++++++++++++++
 MANIFEST                    |  1 +
 Makefile.PL                 |  2 +-
 3 files changed, 71 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/lei-index.pod

diff --git a/Documentation/lei-index.pod b/Documentation/lei-index.pod
new file mode 100644
index 00000000..bd125bcc
--- /dev/null
+++ b/Documentation/lei-index.pod
@@ -0,0 +1,69 @@
+=head1 NAME
+
+lei-index - index messages without importing them into lei/store
+
+=head1 SYNOPSIS
+
+lei index [OPTIONS] FOLDER
+
+lei index [OPTIONS] --stdin
+
+=head1 DESCRIPTION
+
+Similar to L<lei-import(1)>, but does not store a copy of
+messages into C<lei/store>.
+
+This command only makes sense for messages stored in Maildir
+folders.  Other folder types may be supported in the future
+(they can all be indexed, but the message isn't automatically
+retrieved by L<lei-q(1)> or L<lei-lcat(1)>).
+
+Combined with L<lei-q(1)>, C<lei index> allows Maildir users to
+have similar functionality to L<mairix(1)> by not duplicating
+messages into C<lei/store>.
+
+=head1 OPTIONS
+
+=over
+
+=item -
+
+=item --stdin
+
+Read input from standard input.  This is the default if standard
+input is a pipe or regular file and there are no arguments on
+the command-line.
+
+=item -F MAIL_FORMAT
+
+=item --in-format=MAIL_FORMAT
+
+Message input format: C<eml>, C<maildir>, C<imap>, C<imaps>, C<nntp>,
+C<nntps>, C<mboxrd>, C<mboxcl2>, C<mboxcl>, or C<mboxo>.
+
+Default: C<eml> when reading from stdin
+
+=item -q
+
+=item --quiet
+
+Suppress feedback messages.
+
+=back
+
+=head1 CONTACT
+
+Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
+
+The mail archives are hosted at L<https://public-inbox.org/meta/> and
+L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
+
+=head1 COPYRIGHT
+
+Copyright all contributors L<mailto:meta@public-inbox.org>
+
+License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
+
+=head1 SEE ALSO
+
+L<lei-store-format(5)>, L<lei-import(1)>
diff --git a/MANIFEST b/MANIFEST
index 531f8c46..c0e3e855 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -31,6 +31,7 @@ Documentation/lei-edit-search.pod
 Documentation/lei-forget-external.pod
 Documentation/lei-forget-search.pod
 Documentation/lei-import.pod
+Documentation/lei-index.pod
 Documentation/lei-init.pod
 Documentation/lei-lcat.pod
 Documentation/lei-ls-external.pod
diff --git a/Makefile.PL b/Makefile.PL
index 82b50543..bfabb171 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -47,7 +47,7 @@ $v->{-m1} = [ map {
 	qw(
 	lei-add-external lei-blob lei-config lei-convert lei-edit-search
 	lei-daemon-kill lei-daemon-pid lei-forget-external lei-forget-search
-	lei-import lei-init lei-lcat lei-ls-external lei-ls-label
+	lei-import lei-index lei-init lei-lcat lei-ls-external lei-ls-label
 	lei-ls-mail-sync lei-ls-search lei-p2q lei-q lei-rediff lei-rm lei-tag
 	lei-up)];
 $v->{-m5} = [ qw(public-inbox-config public-inbox-v1-format

^ permalink raw reply related	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2021-09-10  9:08 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-10  9:08 [PATCH 0/4] lei: some net-related things Eric Wong
2021-09-10  9:08 ` [PATCH 1/4] lei_query: fix comment about %lei2curl commands Eric Wong
2021-09-10  9:08 ` [PATCH 2/4] lei: split out @net_opt for curl/torsocks use Eric Wong
2021-09-10  9:08 ` [PATCH 3/4] lei: do not read ~/.netrc by default Eric Wong
2021-09-10  9:08 ` [PATCH 4/4] doc: lei-index manpage Eric Wong

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).