Expect more on the way. As always some bugs and fixes were found while attempting to write docs :x Eric Wong (8): xt/net_writer_imap: test "lei convert" w/ IMAP source lei convert: use "--output" in failure message doc: lei-convert: various updates and cleanups doc: lei blob: wording fixups, describe --remote lei convert: remove redundant input_net_cb doc: lei-add-watch: add warning about unreliability lei sucks: show nproc in CPU info lei rm: move generic input_maildir_cb to LeiInput parent class Documentation/lei-add-watch.pod | 8 ++++++-- Documentation/lei-blob.pod | 12 ++++++++---- Documentation/lei-convert.pod | 21 +++++++++------------ lib/PublicInbox/LeiConvert.pm | 8 ++------ lib/PublicInbox/LeiInput.pm | 5 +++++ lib/PublicInbox/LeiRm.pm | 5 ----- lib/PublicInbox/LeiSucks.pm | 4 +++- t/lei-convert.t | 10 ++++++++++ xt/net_writer-imap.t | 7 +++++++ 9 files changed, 50 insertions(+), 30 deletions(-)
I just did a double-take and nearly thought authentication was broken while reading LeiConvert.pm. Add a comment in LeiConvert.pm to clarify things, too. --- lib/PublicInbox/LeiConvert.pm | 1 + xt/net_writer-imap.t | 7 +++++++ 2 files changed, 8 insertions(+) diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm index 68fc7c0ba2b0..424eab8e22e2 100644 --- a/lib/PublicInbox/LeiConvert.pm +++ b/lib/PublicInbox/LeiConvert.pm @@ -55,6 +55,7 @@ sub lei_convert { # the main "lei convert" method my $devfd = $lei->path_to_fd($ovv->{dst}) // return; $lei->{opt}->{augment} = 1 if $devfd < 0; $self->prepare_inputs($lei, \@inputs) or return; + # n.b. {net} {auth} is handled by l2m worker my ($op_c, $ops) = $lei->workers_start($self, 1); $lei->{wq1} = $self; $self->wq_io_do('process_inputs', []); diff --git a/xt/net_writer-imap.t b/xt/net_writer-imap.t index afa4bcc3e881..333e0e3b7ee0 100644 --- a/xt/net_writer-imap.t +++ b/xt/net_writer-imap.t @@ -137,6 +137,13 @@ test_lei(sub { is_deeply($res->[0]->[1], $plack_qp_eml, 'lei q wrote expected result'); + my $mdir = "$ENV{HOME}/t.mdir"; + lei_ok 'convert', $folder_url, '-o', $mdir; + my @mdfiles = glob("$mdir/*/*"); + is(scalar(@mdfiles), 1, '1 message from IMAP => Maildir conversion'); + is_deeply(eml_load($mdfiles[0]), $plack_qp_eml, + 'conversion from IMAP to Maildir'); + lei_ok qw(q f:matz -a -o), $folder_url; $nwr->imap_each($folder_uri, $imap_slurp_all, my $aug = []); is(scalar(@$aug), 2, '2 results after augment') or diag explain($aug);
The extra dashes should help users find the correct option more easily. --- lib/PublicInbox/LeiConvert.pm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm index 424eab8e22e2..623599aef883 100644 --- a/lib/PublicInbox/LeiConvert.pm +++ b/lib/PublicInbox/LeiConvert.pm @@ -51,7 +51,7 @@ sub lei_convert { # the main "lei convert" method my $self = bless {}, __PACKAGE__; my $ovv = PublicInbox::LeiOverview->new($lei, 'out-format'); $lei->{l2m} or return - $lei->fail("output not specified or is not a mail destination"); + $lei->fail('--output unspecified or is not a mail destination'); my $devfd = $lei->path_to_fd($ovv->{dst}) // return; $lei->{opt}->{augment} = 1 if $devfd < 0; $self->prepare_inputs($lei, \@inputs) or return;
Note that "-o OUTPUT" is required in the synopsis. Leave out "eml:" for now since it doesn't work as an output and I doubt anybody would use it as a prefix, and it's not really useful. --no-import-remote is also not accepted by convert, since it doesn't touch lei/store at all. --- Documentation/lei-convert.pod | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-) diff --git a/Documentation/lei-convert.pod b/Documentation/lei-convert.pod index 750ba54ffa1a..c113db18b1e4 100644 --- a/Documentation/lei-convert.pod +++ b/Documentation/lei-convert.pod @@ -1,23 +1,23 @@ =head1 NAME -lei-convert - one-time conversion from URL or filesystem to another format +lei-convert - one-time conversion from one mail format to another =head1 SYNOPSIS -lei convert [OPTIONS] LOCATION +lei convert -o OUTPUT [OPTIONS] LOCATION -lei convert [OPTIONS] (--stdin|-) +lei convert -o OUTPUT [OPTIONS] (--stdin|-) =head1 DESCRIPTION Convert messages to another format. C<LOCATION> is a source of -messages: a directory (Maildir), a file, or a URL (C<imap://>, -C<imaps://>, C<nntp://>, or C<nntps://>). URLs requiring -authentication use L<git-credential(1)> to +messages: a directory (Maildir), a file (various mbox), or a URL +(C<imap://>, C<imaps://>, C<nntp://>, or C<nntps://>). URLs +requiring authentication use L<git-credential(1)> to fill in the username and password. For a regular file, the location must have a C<E<lt>formatE<gt>:> -prefix specifying one of the following formats: C<eml>, C<mboxrd>, +prefix specifying one of the following formats: C<mboxrd>, C<mboxcl2>, C<mboxcl>, or C<mboxo>. =head1 OPTIONS @@ -44,8 +44,6 @@ L<lei-q(1)>. =item --no-kw -=item --no-import-remote - =item --torsocks=auto|no|yes =item --no-torsocks @@ -63,11 +61,10 @@ and L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta =head1 COPYRIGHT -Copyright 2021 all contributors L<mailto:meta@public-inbox.org> +Copyright all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> - =head1 SEE ALSO -L<lei-add-external(1)> +L<lei-mail-formats(5)>
There's no current way to retrieve blobs by OID directly from remote externals. Maybe the $INBOX_NAME/$OID/s/raw.eml endpoint could be overloaded for that. --- Documentation/lei-blob.pod | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/Documentation/lei-blob.pod b/Documentation/lei-blob.pod index 429d206e358e..e401bb4711a6 100644 --- a/Documentation/lei-blob.pod +++ b/Documentation/lei-blob.pod @@ -9,9 +9,10 @@ lei blob [OPTIONS] OID =head1 DESCRIPTION Display a git blob. The blob may correspond to a message from the -local store, an existing blob in the current repository, or a -not-yet-created blob in the current git project repository (if any) -that can be reconstructed from a message. +local store, any local external, or blobs associated with a +project git repository (if run from a git (working) directory). +For blobs which do not exist, it will attempt to recreate the blob +using patch emails. =head1 OPTIONS @@ -64,7 +65,10 @@ L<lei-q(1)>. =over -=item --[no-]remote +=item --remote + +Remote externals only get queried when the blob needs to be +reconstructed from patch emails. =item --no-local
Use the one provided by the LeiInput parent class. --- lib/PublicInbox/LeiConvert.pm | 5 ----- 1 file changed, 5 deletions(-) diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm index 623599aef883..906f30268344 100644 --- a/lib/PublicInbox/LeiConvert.pm +++ b/lib/PublicInbox/LeiConvert.pm @@ -23,11 +23,6 @@ sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh $self->{wcb}->(undef, {}, $eml); } -sub input_net_cb { # callback for ->imap_each, ->nntp_each - my (undef, undef, $kw, $eml, $self) = @_; # @_[0,1]: url + uid ignored - $self->{wcb}->(undef, { kw => $kw }, $eml); -} - sub input_maildir_cb { my (undef, $kw, $eml, $self) = @_; # $_[0] $filename ignored $self->{wcb}->(undef, { kw => $kw }, $eml);
This needs work at some point in the future. --- Documentation/lei-add-watch.pod | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/Documentation/lei-add-watch.pod b/Documentation/lei-add-watch.pod index 609846181578..1b48b43b589c 100644 --- a/Documentation/lei-add-watch.pod +++ b/Documentation/lei-add-watch.pod @@ -11,6 +11,11 @@ lei add-watch [OPTIONS] LOCATION [LOCATION...] Tell lei to watch C<LOCATION> for new messages and flag changes. Currently only Maildir locations are supported. +WARNING: watches are not always reliable, occasional use +of L<lei-index(1)> and L<lei-refresh-mail-sync(1)> is recommended +if L<lei-daemon(8)> crashes or needs to be restarted. This will +be improved in the future. + =for comment TODO: Document --state? Believe valid values are pause, import-ro, @@ -23,11 +28,10 @@ L<http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/> =head1 COPYRIGHT -Copyright 2021 all contributors L<mailto:meta@public-inbox.org> +Copyright all contributors L<mailto:meta@public-inbox.org> License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt> - =head1 SEE ALSO L<lei-ls-watch(1)>, L<lei-rm-watch(1)>
Some bugs are triggered with more CPUs, some with 1 CPU. --- lib/PublicInbox/LeiSucks.pm | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/PublicInbox/LeiSucks.pm b/lib/PublicInbox/LeiSucks.pm index e832f95e0736..8e866fc96655 100644 --- a/lib/PublicInbox/LeiSucks.pm +++ b/lib/PublicInbox/LeiSucks.pm @@ -11,6 +11,7 @@ use Digest::SHA (); use Config; use POSIX (); use PublicInbox::Config; +use PublicInbox::IPC; sub lei_sucks { my ($lei, @argv) = @_; @@ -22,9 +23,10 @@ sub lei_sucks { } eval { require PublicInbox }; my $pi_ver = eval('$PublicInbox::VERSION') // '(???)'; + my $nproc = PublicInbox::IPC::detect_nproc() // '?'; my @out = ("lei $pi_ver\n", "perl $Config{version} / $os $rel / $mac ". - "ptrsize=$Config{ptrsize}\n"); + "ptrsize=$Config{ptrsize} nproc=$nproc\n"); chomp(my $gv = `git --version` || "git missing"); $gv =~ s/ version / /; my $json = ref(PublicInbox::Config->json);
It's not much of a savings, right now, but maybe it can be in the future. I wanted to eliminate the "lei convert" one, too, but convert needs to preserve keywords which isn't possible with the generic fallback, so new tests were written for convert, instead. --- lib/PublicInbox/LeiInput.pm | 5 +++++ lib/PublicInbox/LeiRm.pm | 5 ----- t/lei-convert.t | 10 ++++++++++ 3 files changed, 15 insertions(+), 5 deletions(-) diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm index 540681e3ff6b..84fc579dc480 100644 --- a/lib/PublicInbox/LeiInput.pm +++ b/lib/PublicInbox/LeiInput.pm @@ -64,6 +64,11 @@ sub input_mbox_cb { # base MboxReader callback $self->input_eml_cb($eml); } +sub input_maildir_cb { + my ($fn, $kw, $eml, $self) = @_; + $self->input_eml_cb($eml); +} + sub input_net_cb { # imap_each, nntp_each cb my ($url, $uid, $kw, $eml, $self) = @_; $self->input_eml_cb($eml); diff --git a/lib/PublicInbox/LeiRm.pm b/lib/PublicInbox/LeiRm.pm index 524c178e3b47..cc1abbff66cb 100644 --- a/lib/PublicInbox/LeiRm.pm +++ b/lib/PublicInbox/LeiRm.pm @@ -13,11 +13,6 @@ sub input_eml_cb { # used by PublicInbox::LeiInput::input_fh $self->{lei}->{sto}->wq_do('remove_eml', $eml); } -sub input_maildir_cb { - my (undef, $kw, $eml, $self) = @_; # $_[0] $filename ignored - input_eml_cb($self, $eml); -} - sub lei_rm { my ($lei, @inputs) = @_; $lei->_lei_store(1)->write_prepare($lei); diff --git a/t/lei-convert.t b/t/lei-convert.t index 0ea860c82189..e1849ff796cd 100644 --- a/t/lei-convert.t +++ b/t/lei-convert.t @@ -115,5 +115,15 @@ test_lei({ tmpdir => $tmpdir }, sub { @bar = (); PublicInbox::MboxReader->mboxrd($fh, sub { push @bar, shift }); is_deeply(\@bar, [ $qp_eml ], 'readed gzipped mboxrd'); + + # Status => Maildir flag => Status round trip + $lei_out =~ s/^Status: O/Status: RO/sm or xbail "`seen' Status"; + $rdr = { 0 => \($in = $lei_out), %$lei_opt }; + lei_ok([qw(convert -F mboxrd -o), "$d/md2"], undef, $rdr); + @md = glob("$d/md2/*/*"); + is(scalar(@md), 1, 'one message'); + like($md[0], qr/:2,S\z/, "`seen' flag set in Maildir"); + lei_ok(qw(convert -o mboxrd:/dev/stdout), "$d/md2"); + like($lei_out, qr/^Status: RO/sm, "`seen' flag preserved"); }); done_testing;