* [PATCH 0/8] lei: docs and cleanups
@ 2021-10-28 11:14 Eric Wong
2021-10-28 11:14 ` [PATCH 1/8] xt/net_writer_imap: test "lei convert" w/ IMAP source Eric Wong
` (7 more replies)
0 siblings, 8 replies; 9+ messages in thread
From: Eric Wong @ 2021-10-28 11:14 UTC (permalink / raw)
To: meta
Expect more on the way. As always some bugs and fixes were
found while attempting to write docs :x
Eric Wong (8):
xt/net_writer_imap: test "lei convert" w/ IMAP source
lei convert: use "--output" in failure message
doc: lei-convert: various updates and cleanups
doc: lei blob: wording fixups, describe --remote
lei convert: remove redundant input_net_cb
doc: lei-add-watch: add warning about unreliability
lei sucks: show nproc in CPU info
lei rm: move generic input_maildir_cb to LeiInput parent class
Documentation/lei-add-watch.pod | 8 ++++++--
Documentation/lei-blob.pod | 12 ++++++++----
Documentation/lei-convert.pod | 21 +++++++++------------
lib/PublicInbox/LeiConvert.pm | 8 ++------
lib/PublicInbox/LeiInput.pm | 5 +++++
lib/PublicInbox/LeiRm.pm | 5 -----
lib/PublicInbox/LeiSucks.pm | 4 +++-
t/lei-convert.t | 10 ++++++++++
xt/net_writer-imap.t | 7 +++++++
9 files changed, 50 insertions(+), 30 deletions(-)
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 1/8] xt/net_writer_imap: test "lei convert" w/ IMAP source
2021-10-28 11:14 [PATCH 0/8] lei: docs and cleanups Eric Wong
@ 2021-10-28 11:14 ` Eric Wong
2021-10-28 11:14 ` [PATCH 2/8] lei convert: use "--output" in failure message Eric Wong
` (6 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-10-28 11:14 UTC (permalink / raw)
To: meta
I just did a double-take and nearly thought authentication
was broken while reading LeiConvert.pm. Add a comment in
LeiConvert.pm to clarify things, too.
---
lib/PublicInbox/LeiConvert.pm | 1 +
xt/net_writer-imap.t | 7 +++++++
2 files changed, 8 insertions(+)
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 68fc7c0ba2b0..424eab8e22e2 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -55,6 +55,7 @@ sub lei_convert { # the main "lei convert" method
my $devfd = $lei->path_to_fd($ovv->{dst}) // return;
$lei->{opt}->{augment} = 1 if $devfd < 0;
$self->prepare_inputs($lei, \@inputs) or return;
+ # n.b. {net} {auth} is handled by l2m worker
my ($op_c, $ops) = $lei->workers_start($self, 1);
$lei->{wq1} = $self;
$self->wq_io_do('process_inputs', []);
diff --git a/xt/net_writer-imap.t b/xt/net_writer-imap.t
index afa4bcc3e881..333e0e3b7ee0 100644
--- a/xt/net_writer-imap.t
+++ b/xt/net_writer-imap.t
@@ -137,6 +137,13 @@ test_lei(sub {
is_deeply($res->[0]->[1], $plack_qp_eml,
'lei q wrote expected result');
+ my $mdir = "$ENV{HOME}/t.mdir";
+ lei_ok 'convert', $folder_url, '-o', $mdir;
+ my @mdfiles = glob("$mdir/*/*");
+ is(scalar(@mdfiles), 1, '1 message from IMAP => Maildir conversion');
+ is_deeply(eml_load($mdfiles[0]), $plack_qp_eml,
+ 'conversion from IMAP to Maildir');
+
lei_ok qw(q f:matz -a -o), $folder_url;
$nwr->imap_each($folder_uri, $imap_slurp_all, my $aug = []);
is(scalar(@$aug), 2, '2 results after augment') or diag explain($aug);
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 2/8] lei convert: use "--output" in failure message
2021-10-28 11:14 [PATCH 0/8] lei: docs and cleanups Eric Wong
2021-10-28 11:14 ` [PATCH 1/8] xt/net_writer_imap: test "lei convert" w/ IMAP source Eric Wong
@ 2021-10-28 11:14 ` Eric Wong
2021-10-28 11:14 ` [PATCH 3/8] doc: lei-convert: various updates and cleanups Eric Wong
` (5 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-10-28 11:14 UTC (permalink / raw)
To: meta
The extra dashes should help users find the correct option
more easily.
---
lib/PublicInbox/LeiConvert.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 424eab8e22e2..623599aef883 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -51,7 +51,7 @@ sub lei_convert { # the main "lei convert" method
my $self = bless {}, __PACKAGE__;
my $ovv = PublicInbox::LeiOverview->new($lei, 'out-format');
$lei->{l2m} or return
- $lei->fail("output not specified or is not a mail destination");
+ $lei->fail('--output unspecified or is not a mail destination');
my $devfd = $lei->path_to_fd($ovv->{dst}) // return;
$lei->{opt}->{augment} = 1 if $devfd < 0;
$self->prepare_inputs($lei, \@inputs) or return;
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 3/8] doc: lei-convert: various updates and cleanups
2021-10-28 11:14 [PATCH 0/8] lei: docs and cleanups Eric Wong
2021-10-28 11:14 ` [PATCH 1/8] xt/net_writer_imap: test "lei convert" w/ IMAP source Eric Wong
2021-10-28 11:14 ` [PATCH 2/8] lei convert: use "--output" in failure message Eric Wong
@ 2021-10-28 11:14 ` Eric Wong
2021-10-28 11:14 ` [PATCH 4/8] doc: lei blob: wording fixups, describe --remote Eric Wong
` (4 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-10-28 11:14 UTC (permalink / raw)
To: meta
Note that "-o OUTPUT" is required in the synopsis.
Leave out "eml:" for now since it doesn't work as an output and
I doubt anybody would use it as a prefix, and it's not really
useful.
--no-import-remote is also not accepted by convert, since it
doesn't touch lei/store at all.
---
Documentation/lei-convert.pod | 21 +++++++++------------
1 file changed, 9 insertions(+), 12 deletions(-)
diff --git a/Documentation/lei-convert.pod b/Documentation/lei-convert.pod
index 750ba54ffa1a..c113db18b1e4 100644
--- a/Documentation/lei-convert.pod
+++ b/Documentation/lei-convert.pod
@@ -1,23 +1,23 @@
=head1 NAME
-lei-convert - one-time conversion from URL or filesystem to another format
+lei-convert - one-time conversion from one mail format to another
=head1 SYNOPSIS
-lei convert [OPTIONS] LOCATION
+lei convert -o OUTPUT [OPTIONS] LOCATION
-lei convert [OPTIONS] (--stdin|-)
+lei convert -o OUTPUT [OPTIONS] (--stdin|-)
=head1 DESCRIPTION
Convert messages to another format. C<LOCATION> is a source of
-messages: a directory (Maildir), a file, or a URL (C<imap://>,
-C<imaps://>, C<nntp://>, or C<nntps://>). URLs requiring
-authentication use L<git-credential(1)> to
+messages: a directory (Maildir), a file (various mbox), or a URL
+(C<imap://>, C<imaps://>, C<nntp://>, or C<nntps://>). URLs
+requiring authentication use L<git-credential(1)> to
fill in the username and password.
For a regular file, the location must have a C<E<lt>formatE<gt>:>
-prefix specifying one of the following formats: C<eml>, C<mboxrd>,
+prefix specifying one of the following formats: C<mboxrd>,
C<mboxcl2>, C<mboxcl>, or C<mboxo>.
=head1 OPTIONS
@@ -44,8 +44,6 @@ L<lei-q(1)>.
=item --no-kw
-=item --no-import-remote
-
=item --torsocks=auto|no|yes
=item --no-torsocks
@@ -63,11 +61,10 @@ and L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta
=head1 COPYRIGHT
-Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+Copyright all contributors L<mailto:meta@public-inbox.org>
License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
-
=head1 SEE ALSO
-L<lei-add-external(1)>
+L<lei-mail-formats(5)>
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 4/8] doc: lei blob: wording fixups, describe --remote
2021-10-28 11:14 [PATCH 0/8] lei: docs and cleanups Eric Wong
` (2 preceding siblings ...)
2021-10-28 11:14 ` [PATCH 3/8] doc: lei-convert: various updates and cleanups Eric Wong
@ 2021-10-28 11:14 ` Eric Wong
2021-10-28 11:14 ` [PATCH 5/8] lei convert: remove redundant input_net_cb Eric Wong
` (3 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-10-28 11:14 UTC (permalink / raw)
To: meta
There's no current way to retrieve blobs by OID directly
from remote externals. Maybe the $INBOX_NAME/$OID/s/raw.eml
endpoint could be overloaded for that.
---
Documentation/lei-blob.pod | 12 ++++++++----
1 file changed, 8 insertions(+), 4 deletions(-)
diff --git a/Documentation/lei-blob.pod b/Documentation/lei-blob.pod
index 429d206e358e..e401bb4711a6 100644
--- a/Documentation/lei-blob.pod
+++ b/Documentation/lei-blob.pod
@@ -9,9 +9,10 @@ lei blob [OPTIONS] OID
=head1 DESCRIPTION
Display a git blob. The blob may correspond to a message from the
-local store, an existing blob in the current repository, or a
-not-yet-created blob in the current git project repository (if any)
-that can be reconstructed from a message.
+local store, any local external, or blobs associated with a
+project git repository (if run from a git (working) directory).
+For blobs which do not exist, it will attempt to recreate the blob
+using patch emails.
=head1 OPTIONS
@@ -64,7 +65,10 @@ L<lei-q(1)>.
=over
-=item --[no-]remote
+=item --remote
+
+Remote externals only get queried when the blob needs to be
+reconstructed from patch emails.
=item --no-local
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 5/8] lei convert: remove redundant input_net_cb
2021-10-28 11:14 [PATCH 0/8] lei: docs and cleanups Eric Wong
` (3 preceding siblings ...)
2021-10-28 11:14 ` [PATCH 4/8] doc: lei blob: wording fixups, describe --remote Eric Wong
@ 2021-10-28 11:14 ` Eric Wong
2021-10-28 11:14 ` [PATCH 6/8] doc: lei-add-watch: add warning about unreliability Eric Wong
` (2 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-10-28 11:14 UTC (permalink / raw)
To: meta
Use the one provided by the LeiInput parent class.
---
lib/PublicInbox/LeiConvert.pm | 5 -----
1 file changed, 5 deletions(-)
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 623599aef883..906f30268344 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -23,11 +23,6 @@ sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
$self->{wcb}->(undef, {}, $eml);
}
-sub input_net_cb { # callback for ->imap_each, ->nntp_each
- my (undef, undef, $kw, $eml, $self) = @_; # @_[0,1]: url + uid ignored
- $self->{wcb}->(undef, { kw => $kw }, $eml);
-}
-
sub input_maildir_cb {
my (undef, $kw, $eml, $self) = @_; # $_[0] $filename ignored
$self->{wcb}->(undef, { kw => $kw }, $eml);
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 6/8] doc: lei-add-watch: add warning about unreliability
2021-10-28 11:14 [PATCH 0/8] lei: docs and cleanups Eric Wong
` (4 preceding siblings ...)
2021-10-28 11:14 ` [PATCH 5/8] lei convert: remove redundant input_net_cb Eric Wong
@ 2021-10-28 11:14 ` Eric Wong
2021-10-28 11:15 ` [PATCH 7/8] lei sucks: show nproc in CPU info Eric Wong
2021-10-28 11:15 ` [PATCH 8/8] lei rm: move generic input_maildir_cb to LeiInput parent class Eric Wong
7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-10-28 11:14 UTC (permalink / raw)
To: meta
This needs work at some point in the future.
---
Documentation/lei-add-watch.pod | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/Documentation/lei-add-watch.pod b/Documentation/lei-add-watch.pod
index 609846181578..1b48b43b589c 100644
--- a/Documentation/lei-add-watch.pod
+++ b/Documentation/lei-add-watch.pod
@@ -11,6 +11,11 @@ lei add-watch [OPTIONS] LOCATION [LOCATION...]
Tell lei to watch C<LOCATION> for new messages and flag changes.
Currently only Maildir locations are supported.
+WARNING: watches are not always reliable, occasional use
+of L<lei-index(1)> and L<lei-refresh-mail-sync(1)> is recommended
+if L<lei-daemon(8)> crashes or needs to be restarted. This will
+be improved in the future.
+
=for comment
TODO: Document --state? Believe valid values are pause, import-ro,
@@ -23,11 +28,10 @@ L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
=head1 COPYRIGHT
-Copyright 2021 all contributors L<mailto:meta@public-inbox.org>
+Copyright all contributors L<mailto:meta@public-inbox.org>
License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
-
=head1 SEE ALSO
L<lei-ls-watch(1)>, L<lei-rm-watch(1)>
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 7/8] lei sucks: show nproc in CPU info
2021-10-28 11:14 [PATCH 0/8] lei: docs and cleanups Eric Wong
` (5 preceding siblings ...)
2021-10-28 11:14 ` [PATCH 6/8] doc: lei-add-watch: add warning about unreliability Eric Wong
@ 2021-10-28 11:15 ` Eric Wong
2021-10-28 11:15 ` [PATCH 8/8] lei rm: move generic input_maildir_cb to LeiInput parent class Eric Wong
7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-10-28 11:15 UTC (permalink / raw)
To: meta
Some bugs are triggered with more CPUs, some with 1 CPU.
---
lib/PublicInbox/LeiSucks.pm | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiSucks.pm b/lib/PublicInbox/LeiSucks.pm
index e832f95e0736..8e866fc96655 100644
--- a/lib/PublicInbox/LeiSucks.pm
+++ b/lib/PublicInbox/LeiSucks.pm
@@ -11,6 +11,7 @@ use Digest::SHA ();
use Config;
use POSIX ();
use PublicInbox::Config;
+use PublicInbox::IPC;
sub lei_sucks {
my ($lei, @argv) = @_;
@@ -22,9 +23,10 @@ sub lei_sucks {
}
eval { require PublicInbox };
my $pi_ver = eval('$PublicInbox::VERSION') // '(???)';
+ my $nproc = PublicInbox::IPC::detect_nproc() // '?';
my @out = ("lei $pi_ver\n",
"perl $Config{version} / $os $rel / $mac ".
- "ptrsize=$Config{ptrsize}\n");
+ "ptrsize=$Config{ptrsize} nproc=$nproc\n");
chomp(my $gv = `git --version` || "git missing");
$gv =~ s/ version / /;
my $json = ref(PublicInbox::Config->json);
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 8/8] lei rm: move generic input_maildir_cb to LeiInput parent class
2021-10-28 11:14 [PATCH 0/8] lei: docs and cleanups Eric Wong
` (6 preceding siblings ...)
2021-10-28 11:15 ` [PATCH 7/8] lei sucks: show nproc in CPU info Eric Wong
@ 2021-10-28 11:15 ` Eric Wong
7 siblings, 0 replies; 9+ messages in thread
From: Eric Wong @ 2021-10-28 11:15 UTC (permalink / raw)
To: meta
It's not much of a savings, right now, but maybe it can be in the
future. I wanted to eliminate the "lei convert" one, too, but
convert needs to preserve keywords which isn't possible with the
generic fallback, so new tests were written for convert, instead.
---
lib/PublicInbox/LeiInput.pm | 5 +++++
lib/PublicInbox/LeiRm.pm | 5 -----
t/lei-convert.t | 10 ++++++++++
3 files changed, 15 insertions(+), 5 deletions(-)
diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm
index 540681e3ff6b..84fc579dc480 100644
--- a/lib/PublicInbox/LeiInput.pm
+++ b/lib/PublicInbox/LeiInput.pm
@@ -64,6 +64,11 @@ sub input_mbox_cb { # base MboxReader callback
$self->input_eml_cb($eml);
}
+sub input_maildir_cb {
+ my ($fn, $kw, $eml, $self) = @_;
+ $self->input_eml_cb($eml);
+}
+
sub input_net_cb { # imap_each, nntp_each cb
my ($url, $uid, $kw, $eml, $self) = @_;
$self->input_eml_cb($eml);
diff --git a/lib/PublicInbox/LeiRm.pm b/lib/PublicInbox/LeiRm.pm
index 524c178e3b47..cc1abbff66cb 100644
--- a/lib/PublicInbox/LeiRm.pm
+++ b/lib/PublicInbox/LeiRm.pm
@@ -13,11 +13,6 @@ sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh
$self->{lei}->{sto}->wq_do('remove_eml', $eml);
}
-sub input_maildir_cb {
- my (undef, $kw, $eml, $self) = @_; # $_[0] $filename ignored
- input_eml_cb($self, $eml);
-}
-
sub lei_rm {
my ($lei, @inputs) = @_;
$lei->_lei_store(1)->write_prepare($lei);
diff --git a/t/lei-convert.t b/t/lei-convert.t
index 0ea860c82189..e1849ff796cd 100644
--- a/t/lei-convert.t
+++ b/t/lei-convert.t
@@ -115,5 +115,15 @@ test_lei({ tmpdir => $tmpdir }, sub {
@bar = ();
PublicInbox::MboxReader->mboxrd($fh, sub { push @bar, shift });
is_deeply(\@bar, [ $qp_eml ], 'readed gzipped mboxrd');
+
+ # Status => Maildir flag => Status round trip
+ $lei_out =~ s/^Status: O/Status: RO/sm or xbail "`seen' Status";
+ $rdr = { 0 => \($in = $lei_out), %$lei_opt };
+ lei_ok([qw(convert -F mboxrd -o), "$d/md2"], undef, $rdr);
+ @md = glob("$d/md2/*/*");
+ is(scalar(@md), 1, 'one message');
+ like($md[0], qr/:2,S\z/, "`seen' flag set in Maildir");
+ lei_ok(qw(convert -o mboxrd:/dev/stdout), "$d/md2");
+ like($lei_out, qr/^Status: RO/sm, "`seen' flag preserved");
});
done_testing;
^ permalink raw reply related [flat|nested] 9+ messages in thread
end of thread, other threads:[~2021-10-28 11:15 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-10-28 11:14 [PATCH 0/8] lei: docs and cleanups Eric Wong
2021-10-28 11:14 ` [PATCH 1/8] xt/net_writer_imap: test "lei convert" w/ IMAP source Eric Wong
2021-10-28 11:14 ` [PATCH 2/8] lei convert: use "--output" in failure message Eric Wong
2021-10-28 11:14 ` [PATCH 3/8] doc: lei-convert: various updates and cleanups Eric Wong
2021-10-28 11:14 ` [PATCH 4/8] doc: lei blob: wording fixups, describe --remote Eric Wong
2021-10-28 11:14 ` [PATCH 5/8] lei convert: remove redundant input_net_cb Eric Wong
2021-10-28 11:14 ` [PATCH 6/8] doc: lei-add-watch: add warning about unreliability Eric Wong
2021-10-28 11:15 ` [PATCH 7/8] lei sucks: show nproc in CPU info Eric Wong
2021-10-28 11:15 ` [PATCH 8/8] lei rm: move generic input_maildir_cb to LeiInput parent class Eric Wong
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).