* [PATCH 00/12] lei blob and some yak-shaving
@ 2021-03-28 9:01 Eric Wong
2021-03-28 9:01 ` [PATCH 01/12] lei: simplify PktOp callers Eric Wong
` (11 more replies)
0 siblings, 12 replies; 14+ messages in thread
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
"lei blob" manages to work with HTTPS and HTTP .onion endpoints.
Eric Wong (12):
lei: simplify PktOp callers
lei init: split out into separate file
lei blob: dclose if already failed
lei blob: support --no-mail switch
lei blob: fail early if no git dirs
lei blob: some extra tests
lei help: show "NAME=VALUE" properly for -c
lei blob: flesh out help text
t/lei_store: ensure LeiSearch responds to ->isrch
lei blob: add remote external support
lei: drop coderepo placeholders, submodule TODO
treewide: shorten temporary filename
MANIFEST | 2 +
lib/PublicInbox/LEI.pm | 67 +++++++---------------------
lib/PublicInbox/LeiBlob.pm | 47 +++++++++++++-------
lib/PublicInbox/LeiConvert.pm | 4 +-
lib/PublicInbox/LeiHelp.pm | 2 +-
lib/PublicInbox/LeiImport.pm | 4 +-
lib/PublicInbox/LeiInit.pm | 41 +++++++++++++++++
lib/PublicInbox/LeiMark.pm | 4 +-
lib/PublicInbox/LeiMirror.pm | 4 +-
lib/PublicInbox/LeiOverview.pm | 2 +-
lib/PublicInbox/LeiP2q.pm | 4 +-
lib/PublicInbox/LeiRemote.pm | 81 ++++++++++++++++++++++++++++++++++
lib/PublicInbox/LeiXSearch.pm | 8 ++--
lib/PublicInbox/Msgmap.pm | 2 +-
lib/PublicInbox/PktOp.pm | 20 ++++++---
lib/PublicInbox/SolverGit.pm | 2 +-
lib/PublicInbox/TestCommon.pm | 2 +-
lib/PublicInbox/V2Writable.pm | 6 +--
lib/PublicInbox/Xapcmd.pm | 9 ++--
script/public-inbox-edit | 2 +-
script/public-inbox-init | 2 +-
scripts/ssoma-replay | 2 +-
t/check-www-inbox.perl | 2 +-
t/inbox.t | 2 +-
t/lei_store.t | 1 +
t/nodatacow.t | 2 +-
t/solver_git.t | 40 +++++++++++++++--
27 files changed, 254 insertions(+), 110 deletions(-)
create mode 100644 lib/PublicInbox/LeiInit.pm
create mode 100644 lib/PublicInbox/LeiRemote.pm
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH 01/12] lei: simplify PktOp callers
2021-03-28 9:01 [PATCH 00/12] lei blob and some yak-shaving Eric Wong
@ 2021-03-28 9:01 ` Eric Wong
2021-03-28 9:01 ` [PATCH 02/12] lei init: split out into separate file Eric Wong
` (10 subsequent siblings)
11 siblings, 0 replies; 14+ messages in thread
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
Provide a consistent ->op_wait_event method instead of
forcing callers to loop (or not) at each callsite.
This also avoid a leak possibility by avoiding circular
references.
---
lib/PublicInbox/LEI.pm | 11 +++++------
lib/PublicInbox/LeiBlob.pm | 4 ++--
lib/PublicInbox/LeiConvert.pm | 4 ++--
lib/PublicInbox/LeiImport.pm | 4 ++--
lib/PublicInbox/LeiMark.pm | 4 ++--
lib/PublicInbox/LeiMirror.pm | 4 ++--
lib/PublicInbox/LeiP2q.pm | 4 ++--
lib/PublicInbox/LeiXSearch.pm | 8 +++-----
lib/PublicInbox/PktOp.pm | 20 +++++++++++++++-----
9 files changed, 35 insertions(+), 28 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 478912cd..9cacb142 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -494,11 +494,11 @@ sub _delete_pkt_op { # OnDestroy callback to prevent leaks on die
}
sub pkt_op_pair {
- my ($self, $ops) = @_;
+ my ($self) = @_;
require PublicInbox::OnDestroy;
require PublicInbox::PktOp;
my $end = PublicInbox::OnDestroy->new($$, \&_delete_pkt_op, $self);
- @$self{qw(pkt_op_c pkt_op_p)} = PublicInbox::PktOp->pair($ops);
+ @$self{qw(pkt_op_c pkt_op_p)} = PublicInbox::PktOp->pair;
$end;
}
@@ -512,14 +512,13 @@ sub workers_start {
($ops ? %$ops : ()),
};
$ops->{''} //= [ \&dclose, $lei ];
- my $end = $lei->pkt_op_pair($ops);
+ my $end = $lei->pkt_op_pair;
$wq->wq_workers_start($ident, $jobs, $lei->oldset, { lei => $lei });
delete $lei->{pkt_op_p};
- my $op = delete $lei->{pkt_op_c};
+ my $op_c = delete $lei->{pkt_op_c};
@$end = ();
$lei->event_step_init;
- # oneshot needs $op, daemon-mode uses DS->EventLoop to handle $op
- $lei->{oneshot} ? $op : undef;
+ ($op_c, $ops);
}
sub _help {
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 2facbad3..97747220 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -103,12 +103,12 @@ sub lei_blob {
my $lxs = $lei->lxs_prepare or return;
require PublicInbox::SolverGit;
my $self = bless { lxs => $lxs, oid_b => $blob }, __PACKAGE__;
- my $op = $lei->workers_start($self, 'lei_solve', 1,
+ my ($op_c, $ops) = $lei->workers_start($self, 'lei_solve', 1,
{ '' => [ \&sol_done, $lei ] });
$lei->{sol} = $self;
$self->wq_io_do('do_solve_blob', []);
$self->wq_close(1);
- while ($op && $op->{sock}) { $op->event_step }
+ $op_c->op_wait_event($ops);
}
sub ipc_atfork_child {
diff --git a/lib/PublicInbox/LeiConvert.pm b/lib/PublicInbox/LeiConvert.pm
index 083ecc33..5d0adb14 100644
--- a/lib/PublicInbox/LeiConvert.pm
+++ b/lib/PublicInbox/LeiConvert.pm
@@ -53,11 +53,11 @@ sub lei_convert { # the main "lei convert" method
my $devfd = $lei->path_to_fd($ovv->{dst}) // return;
$lei->{opt}->{augment} = 1 if $devfd < 0;
$self->prepare_inputs($lei, \@inputs) or return;
- my $op = $lei->workers_start($self, 'lei_convert', 1);
+ my ($op_c, $ops) = $lei->workers_start($self, 'lei_convert', 1);
$lei->{cnv} = $self;
$self->wq_io_do('do_convert', []);
$self->wq_close(1);
- while ($op && $op->{sock}) { $op->event_step }
+ $op_c->op_wait_event($ops);
}
sub ipc_atfork_child {
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index 7c5b7d09..803b5cda 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -76,11 +76,11 @@ sub lei_import { # the main "lei import" method
my $ops = { '' => [ \&import_done, $lei ] };
$lei->{auth}->op_merge($ops, $self) if $lei->{auth};
$self->{-wq_nr_workers} = $j // 1; # locked
- my $op = $lei->workers_start($self, 'lei_import', undef, $ops);
+ my ($op_c, undef) = $lei->workers_start($self, 'lei_import', $j, $ops);
$lei->{imp} = $self;
$self->wq_io_do('input_stdin', []) if $self->{0};
net_merge_complete($self) unless $lei->{auth};
- while ($op && $op->{sock}) { $op->event_step }
+ $op_c->op_wait_event($ops);
}
no warnings 'once';
diff --git a/lib/PublicInbox/LeiMark.pm b/lib/PublicInbox/LeiMark.pm
index 34846b84..6e611318 100644
--- a/lib/PublicInbox/LeiMark.pm
+++ b/lib/PublicInbox/LeiMark.pm
@@ -116,11 +116,11 @@ sub lei_mark { # the "lei mark" method
my $ops = { '' => [ \&mark_done, $lei ] };
$lei->{auth}->op_merge($ops, $self) if $lei->{auth};
$self->{vmd_mod} = $vmd_mod;
- my $op = $lei->workers_start($self, 'lei_mark', 1, $ops);
+ my ($op_c, undef) = $lei->workers_start($self, 'lei_mark', 1, $ops);
$lei->{mark} = $self;
$self->wq_io_do('input_stdin', []) if $self->{0};
net_merge_complete($self) unless $lei->{auth};
- while ($op && $op->{sock}) { $op->event_step }
+ $op_c->op_wait_event($ops);
}
sub note_missing {
diff --git a/lib/PublicInbox/LeiMirror.pm b/lib/PublicInbox/LeiMirror.pm
index c83386c6..89574d28 100644
--- a/lib/PublicInbox/LeiMirror.pm
+++ b/lib/PublicInbox/LeiMirror.pm
@@ -282,13 +282,13 @@ sub start {
require PublicInbox::Inbox;
require PublicInbox::Admin;
require PublicInbox::InboxWritable;
- my $op = $lei->workers_start($self, 'lei_mirror', 1, {
+ my ($op, $ops) = $lei->workers_start($self, 'lei_mirror', 1, {
'' => [ \&mirror_done, $lei ]
});
$lei->{mrr} = $self;
$self->wq_io_do('do_mirror', []);
$self->wq_close(1);
- while ($op && $op->{sock}) { $op->event_step }
+ $op->op_wait_event($ops);
}
sub ipc_atfork_child {
diff --git a/lib/PublicInbox/LeiP2q.pm b/lib/PublicInbox/LeiP2q.pm
index 25f63a10..a8a3dd2c 100644
--- a/lib/PublicInbox/LeiP2q.pm
+++ b/lib/PublicInbox/LeiP2q.pm
@@ -185,11 +185,11 @@ sub lei_p2q { # the "lei patch-to-query" entry point
} else {
$self->{input} = $input;
}
- my $op = $lei->workers_start($self, 'lei_p2q', 1);
+ my ($op, $ops) = $lei->workers_start($self, 'lei_p2q', 1);
$lei->{p2q} = $self;
$self->wq_io_do('do_p2q', []);
$self->wq_close(1);
- while ($op && $op->{sock}) { $op->event_step }
+ $op->op_wait_event($ops);
}
sub ipc_atfork_child {
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index b41daffe..1a194f1c 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -427,7 +427,7 @@ sub do_query {
'incr_start_query' => [ \&incr_start_query, $self, $l2m ],
};
$lei->{auth}->op_merge($ops, $l2m) if $l2m && $lei->{auth};
- my $end = $lei->pkt_op_pair($ops);
+ my $end = $lei->pkt_op_pair;
$lei->{1}->autoflush(1);
$lei->start_pager if delete $lei->{need_pager};
$lei->{ovv}->ovv_begin($lei);
@@ -445,7 +445,7 @@ sub do_query {
}
$self->wq_workers_start('lei_xsearch', undef,
$lei->oldset, { lei => $lei });
- my $op = delete $lei->{pkt_op_c};
+ my $op_c = delete $lei->{pkt_op_c};
delete $lei->{pkt_op_p};
@$end = ();
$self->{threads} = $lei->{opt}->{threads};
@@ -455,9 +455,7 @@ sub do_query {
start_query($self);
}
$lei->event_step_init; # wait for shutdowns
- if ($lei->{oneshot}) {
- while ($op->{sock}) { $op->event_step }
- }
+ $op_c->op_wait_event($ops);
}
sub add_uri {
diff --git a/lib/PublicInbox/PktOp.pm b/lib/PublicInbox/PktOp.pm
index 5d8e78ea..c3221735 100644
--- a/lib/PublicInbox/PktOp.pm
+++ b/lib/PublicInbox/PktOp.pm
@@ -16,21 +16,23 @@ use PublicInbox::IPC qw(ipc_freeze ipc_thaw);
our @EXPORT_OK = qw(pkt_do);
sub new {
- my ($cls, $r, $ops) = @_;
- my $self = bless { sock => $r, ops => $ops }, $cls;
+ my ($cls, $r) = @_;
+ my $self = bless { sock => $r }, $cls;
if ($PublicInbox::DS::in_loop) { # iff using DS->EventLoop
$r->blocking(0);
$self->SUPER::new($r, EPOLLIN|EPOLLET);
+ } else {
+ $self->{blocking} = 1;
}
$self;
}
# returns a blessed object as the consumer, and a GLOB/IO for the producer
sub pair {
- my ($cls, $ops) = @_;
+ my ($cls) = @_;
my ($c, $p);
socketpair($c, $p, AF_UNIX, SOCK_SEQPACKET, 0) or die "socketpair: $!";
- (new($cls, $c, $ops), $p);
+ (new($cls, $c), $p);
}
sub pkt_do { # for the producer to trigger event_step in consumer
@@ -41,7 +43,7 @@ sub pkt_do { # for the producer to trigger event_step in consumer
sub close {
my ($self) = @_;
my $c = $self->{sock} or return;
- $c->blocking ? delete($self->{sock}) : $self->SUPER::close;
+ $self->{blocking} ? delete($self->{sock}) : $self->SUPER::close;
}
sub event_step {
@@ -73,4 +75,12 @@ sub event_step {
}
}
+# call this when we're ready to wait on events,
+# returns immediately if non-blocking
+sub op_wait_event {
+ my ($self, $ops) = @_;
+ $self->{ops} = $ops;
+ while ($self->{blocking} && $self->{sock}) { event_step($self) }
+}
+
1;
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 02/12] lei init: split out into separate file
2021-03-28 9:01 [PATCH 00/12] lei blob and some yak-shaving Eric Wong
2021-03-28 9:01 ` [PATCH 01/12] lei: simplify PktOp callers Eric Wong
@ 2021-03-28 9:01 ` Eric Wong
2021-03-28 9:01 ` [PATCH 03/12] lei blob: dclose if already failed Eric Wong
` (9 subsequent siblings)
11 siblings, 0 replies; 14+ messages in thread
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
This is a rarely-needed command, so keep it separate file
so it's easier-to-find and maybe saves a bit of RAM.
---
MANIFEST | 1 +
lib/PublicInbox/LEI.pm | 32 -----------------------------
lib/PublicInbox/LeiInit.pm | 41 ++++++++++++++++++++++++++++++++++++++
3 files changed, 42 insertions(+), 32 deletions(-)
create mode 100644 lib/PublicInbox/LeiInit.pm
diff --git a/MANIFEST b/MANIFEST
index 64b3626f..9048b900 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -188,6 +188,7 @@ lib/PublicInbox/LeiDedupe.pm
lib/PublicInbox/LeiExternal.pm
lib/PublicInbox/LeiHelp.pm
lib/PublicInbox/LeiImport.pm
+lib/PublicInbox/LeiInit.pm
lib/PublicInbox/LeiInput.pm
lib/PublicInbox/LeiLsLabel.pm
lib/PublicInbox/LeiMark.pm
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 9cacb142..fdb0bbcf 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -744,38 +744,6 @@ sub lei_config {
x_it($self, $?) if $?;
}
-sub lei_init {
- my ($self, $dir) = @_;
- my $cfg = _lei_cfg($self, 1);
- my $cur = $cfg->{'leistore.dir'};
- $dir //= store_path($self);
- $dir = rel2abs($self, $dir);
- my @cur = stat($cur) if defined($cur);
- $cur = File::Spec->canonpath($cur // $dir);
- my @dir = stat($dir);
- my $exists = "# leistore.dir=$cur already initialized" if @dir;
- if (@cur) {
- if ($cur eq $dir) {
- _lei_store($self, 1)->done;
- return qerr($self, $exists);
- }
-
- # some folks like symlinks and bind mounts :P
- if (@dir && "@cur[1,0]" eq "@dir[1,0]") {
- lei_config($self, 'leistore.dir', $dir);
- _lei_store($self, 1)->done;
- return qerr($self, "$exists (as $cur)");
- }
- return fail($self, <<"");
-E: leistore.dir=$cur already initialized and it is not $dir
-
- }
- lei_config($self, 'leistore.dir', $dir);
- _lei_store($self, 1)->done;
- $exists //= "# leistore.dir=$dir newly initialized";
- return qerr($self, $exists);
-}
-
sub lei_daemon_pid { puts shift, $$ }
sub lei_daemon_kill {
diff --git a/lib/PublicInbox/LeiInit.pm b/lib/PublicInbox/LeiInit.pm
new file mode 100644
index 00000000..c6c0c01b
--- /dev/null
+++ b/lib/PublicInbox/LeiInit.pm
@@ -0,0 +1,41 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# for the "lei init" command, not sure if it's even needed...
+package PublicInbox::LeiInit;
+use v5.10.1;
+use File::Spec;
+
+sub lei_init {
+ my ($self, $dir) = @_;
+ my $cfg = $self->_lei_cfg(1);
+ my $cur = $cfg->{'leistore.dir'};
+ $dir //= $self->store_path;
+ $dir = $self->rel2abs($dir);
+ my @cur = stat($cur) if defined($cur);
+ $cur = File::Spec->canonpath($cur // $dir);
+ my @dir = stat($dir);
+ my $exists = "# leistore.dir=$cur already initialized" if @dir;
+ if (@cur) {
+ if ($cur eq $dir) {
+ $self->_lei_store(1)->done;
+ return $self->qerr($exists);
+ }
+
+ # some folks like symlinks and bind mounts :P
+ if (@dir && "@cur[1,0]" eq "@dir[1,0]") {
+ $self->lei_config('leistore.dir', $dir);
+ $self->_lei_store(1)->done;
+ return $self->qerr("$exists (as $cur)");
+ }
+ return $self->fail(<<"");
+E: leistore.dir=$cur already initialized and it is not $dir
+
+ }
+ $self->lei_config('leistore.dir', $dir);
+ $self->_lei_store(1)->done;
+ $exists //= "# leistore.dir=$dir newly initialized";
+ $self->qerr($exists);
+}
+
+1;
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 03/12] lei blob: dclose if already failed
2021-03-28 9:01 [PATCH 00/12] lei blob and some yak-shaving Eric Wong
2021-03-28 9:01 ` [PATCH 01/12] lei: simplify PktOp callers Eric Wong
2021-03-28 9:01 ` [PATCH 02/12] lei init: split out into separate file Eric Wong
@ 2021-03-28 9:01 ` Eric Wong
2021-03-28 9:01 ` [PATCH 04/12] lei blob: support --no-mail switch Eric Wong
` (8 subsequent siblings)
11 siblings, 0 replies; 14+ messages in thread
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
We must close the socket to trigger pager exit if blob
reconstruction fails. Not sure how to test this in the
test suite...
---
lib/PublicInbox/LeiBlob.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 97747220..9b4c4f30 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -19,7 +19,7 @@ sub sol_done_wait { # dwaitpid callback
sub sol_done { # EOF callback for main daemon
my ($lei) = @_;
- my $sol = delete $lei->{sol} or return;
+ my $sol = delete $lei->{sol} // return $lei->dclose; # already failed
$sol->wq_wait_old(\&sol_done_wait, $lei);
}
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 04/12] lei blob: support --no-mail switch
2021-03-28 9:01 [PATCH 00/12] lei blob and some yak-shaving Eric Wong
` (2 preceding siblings ...)
2021-03-28 9:01 ` [PATCH 03/12] lei blob: dclose if already failed Eric Wong
@ 2021-03-28 9:01 ` Eric Wong
2021-03-28 9:01 ` [PATCH 05/12] lei blob: fail early if no git dirs Eric Wong
` (7 subsequent siblings)
11 siblings, 0 replies; 14+ messages in thread
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
It's possible for a abbreviated OID to be resolved unambiguously
to an email before we attempt to look at externals via xsearch;
so provide a way for a user to force searching coderepos.
If hints (--oid-a, --path-a, --path-b) are present, we'll
assume --no-mail by default, otherwise we'll assume the
user wants to look through mail for a matching blob.
---
lib/PublicInbox/LEI.pm | 4 ++--
lib/PublicInbox/LeiBlob.pm | 23 +++++++++++++----------
t/solver_git.t | 3 +++
3 files changed, 18 insertions(+), 12 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index fdb0bbcf..bad7fad9 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -136,8 +136,8 @@ our %CMD = ( # sorted in order of importance/use:
import-before! lock=s@ rsyncable alert=s@ mua=s verbose|v+), @c_opt,
opt_dash('limit|n=i', '[0-9]+') ],
-'blob' => [ 'OID', 'display a git blob object, solving if necessary',
- qw(git-dir=s@ cwd! verbose|v+ oid-a|A=s path-a|a=s path-b|b=s),
+'blob' => [ 'OID', 'show a git blob, reconstructing from mail if necessary',
+ qw(git-dir=s@ cwd! verbose|v+ mail! oid-a|A=s path-a|a=s path-b|b=s),
@lxs_opt, @c_opt ],
'add-external' => [ 'LOCATION',
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 9b4c4f30..4bd86253 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -8,7 +8,6 @@ use v5.10.1;
use parent qw(PublicInbox::IPC);
use PublicInbox::Spawn qw(spawn popen_rd);
use PublicInbox::DS;
-use PublicInbox::Eml;
sub sol_done_wait { # dwaitpid callback
my ($arg, $pid) = @_;
@@ -85,18 +84,22 @@ sub do_solve_blob { # via wq_do
sub lei_blob {
my ($lei, $blob) = @_;
$lei->start_pager if -t $lei->{1};
+ my $opt = $lei->{opt};
+ my $has_hints = grep(defined, @$opt{qw(oid-a path-a path-b)});
- # first, see if it's a blob returned by "lei q" JSON output:
- my $rdr = { 1 => $lei->{1} };
- open $rdr->{2}, '>', '/dev/null' or die "open: $!";
- my $cmd = [ 'git', '--git-dir='.$lei->ale->git->{git_dir},
- 'cat-file', 'blob', $blob ];
- waitpid(spawn($cmd, $lei->{env}, $rdr), 0);
- return if $? == 0;
+ # first, see if it's a blob returned by "lei q" JSON output:k
+ if ($opt->{mail} // ($has_hints ? 0 : 1)) {
+ my $rdr = { 1 => $lei->{1} };
+ open $rdr->{2}, '>', '/dev/null' or die "open: $!";
+ my $cmd = [ 'git', '--git-dir='.$lei->ale->git->{git_dir},
+ 'cat-file', 'blob', $blob ];
+ waitpid(spawn($cmd, $lei->{env}, $rdr), 0);
+ return if $? == 0;
+ }
# maybe it's a non-email (code) blob from a coderepo
- my $git_dirs = $lei->{opt}->{'git-dir'} //= [];
- if ($lei->{opt}->{'cwd'} //= 1) {
+ my $git_dirs = $opt->{'git-dir'} //= [];
+ if ($opt->{'cwd'} // 1) {
my $cgd = get_git_dir('.');
unshift(@$git_dirs, $cgd) if defined $cgd;
}
diff --git a/t/solver_git.t b/t/solver_git.t
index 22714ae5..7bf3ba21 100644
--- a/t/solver_git.t
+++ b/t/solver_git.t
@@ -34,6 +34,9 @@ test_lei({tmpdir => $tmpdir}, sub {
lei_ok('blob', '69df7d5', '-I', $ibx->{inboxdir});
is(sha1_hex("blob ".length($lei_out)."\0".$lei_out),
$expect, 'blob contents output');
+ my $prev = $lei_out;
+ lei_ok(qw(blob --no-mail 69df7d5 -I), $ibx->{inboxdir});
+ is($lei_out, $prev, '--no-mail works');
# fallbacks
lei_ok('blob', $v1_0_0_tag, '-I', $ibx->{inboxdir});
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 05/12] lei blob: fail early if no git dirs
2021-03-28 9:01 [PATCH 00/12] lei blob and some yak-shaving Eric Wong
` (3 preceding siblings ...)
2021-03-28 9:01 ` [PATCH 04/12] lei blob: support --no-mail switch Eric Wong
@ 2021-03-28 9:01 ` Eric Wong
2021-03-28 9:01 ` [PATCH 06/12] lei blob: some extra tests Eric Wong
` (6 subsequent siblings)
11 siblings, 0 replies; 14+ messages in thread
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
This avoids triggering a "BUG:" message in the solver code.
---
lib/PublicInbox/LeiBlob.pm | 1 +
1 file changed, 1 insertion(+)
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 4bd86253..f44d8af1 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -103,6 +103,7 @@ sub lei_blob {
my $cgd = get_git_dir('.');
unshift(@$git_dirs, $cgd) if defined $cgd;
}
+ return $lei->fail('no --git-dir to try') unless @$git_dirs;
my $lxs = $lei->lxs_prepare or return;
require PublicInbox::SolverGit;
my $self = bless { lxs => $lxs, oid_b => $blob }, __PACKAGE__;
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 06/12] lei blob: some extra tests
2021-03-28 9:01 [PATCH 00/12] lei blob and some yak-shaving Eric Wong
` (4 preceding siblings ...)
2021-03-28 9:01 ` [PATCH 05/12] lei blob: fail early if no git dirs Eric Wong
@ 2021-03-28 9:01 ` Eric Wong
2021-03-28 9:01 ` [PATCH 07/12] lei help: show "NAME=VALUE" properly for -c Eric Wong
` (5 subsequent siblings)
11 siblings, 0 replies; 14+ messages in thread
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
Most of it already gets tested since most of the logic is in
SolverGit, but make sure it's all wired up properly to lei.
---
t/solver_git.t | 21 ++++++++++++++++++++-
1 file changed, 20 insertions(+), 1 deletion(-)
diff --git a/t/solver_git.t b/t/solver_git.t
index 7bf3ba21..6d4b93c7 100644
--- a/t/solver_git.t
+++ b/t/solver_git.t
@@ -29,6 +29,7 @@ my $ibx = create_inbox 'v2', version => 2,
my $v1_0_0_tag = 'cb7c42b1e15577ed2215356a2bf925aef59cdd8d';
my $v1_0_0_tag_short = substr($v1_0_0_tag, 0, 16);
my $expect = '69df7d565d49fbaaeb0a067910f03dc22cd52bd0';
+my $non_existent = 'ee5e32211bf62ab6531bdf39b84b6920d0b6775a';
test_lei({tmpdir => $tmpdir}, sub {
lei_ok('blob', '69df7d5', '-I', $ibx->{inboxdir});
@@ -37,6 +38,25 @@ test_lei({tmpdir => $tmpdir}, sub {
my $prev = $lei_out;
lei_ok(qw(blob --no-mail 69df7d5 -I), $ibx->{inboxdir});
is($lei_out, $prev, '--no-mail works');
+ ok(!lei(qw(blob -I), $ibx->{inboxdir}, $non_existent),
+ 'non-existent blob fails');
+ SKIP: {
+ skip '/.git exists', 1 if -e '/.git';
+ require PublicInbox::OnDestroy;
+ opendir my $dh, '.' or xbail "opendir: $!";
+ my $end = PublicInbox::OnDestroy->new($$, sub {
+ chdir $dh or xbail "chdir: $!";
+ });
+ lei_ok(qw(-C / blob 69df7d5 -I), $ibx->{inboxdir},
+ "--git-dir=$git_dir");
+ is($lei_out, $prev, '--git-dir works');
+
+ ok(!lei(qw(-C / blob --no-cwd 69df7d5 -I), $ibx->{inboxdir}),
+ '--no-cwd works');
+
+ ok(!lei(qw(-C / blob -I), $ibx->{inboxdir}, $non_existent),
+ 'non-existent blob fails');
+ }
# fallbacks
lei_ok('blob', $v1_0_0_tag, '-I', $ibx->{inboxdir});
@@ -163,7 +183,6 @@ EOF
close $cfgfh or die;
my $cfg = PublicInbox::Config->new($cfgpath);
my $www = PublicInbox::WWW->new($cfg);
- my $non_existent = 'ee5e32211bf62ab6531bdf39b84b6920d0b6775a';
my $client = sub {
my ($cb) = @_;
my $mid = '20190401081523.16213-1-BOFH@YHBT.net';
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 07/12] lei help: show "NAME=VALUE" properly for -c
2021-03-28 9:01 [PATCH 00/12] lei blob and some yak-shaving Eric Wong
` (5 preceding siblings ...)
2021-03-28 9:01 ` [PATCH 06/12] lei blob: some extra tests Eric Wong
@ 2021-03-28 9:01 ` Eric Wong
2021-03-28 9:01 ` [PATCH 08/12] lei blob: flesh out help text Eric Wong
` (4 subsequent siblings)
11 siblings, 0 replies; 14+ messages in thread
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
---
lib/PublicInbox/LeiHelp.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiHelp.pm b/lib/PublicInbox/LeiHelp.pm
index 9c1b30a1..fa0e7866 100644
--- a/lib/PublicInbox/LeiHelp.pm
+++ b/lib/PublicInbox/LeiHelp.pm
@@ -49,7 +49,7 @@ sub call {
length($_) > 1 ? push(@l, "--$_") : push(@s, "-$_");
}
if (!scalar(@vals)) { # no args 'threads|t'
- } elsif ($arg_vals =~ s/\A([A-Z_]+)\b//) { # "NAME"
+ } elsif ($arg_vals =~ s/\A([A-Z_=]+)\b//) { # "NAME"
$vals[1] = $1;
} else {
$vals[1] = uc(substr($l[0], 2)); # "--type" => "TYPE"
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 08/12] lei blob: flesh out help text
2021-03-28 9:01 [PATCH 00/12] lei blob and some yak-shaving Eric Wong
` (6 preceding siblings ...)
2021-03-28 9:01 ` [PATCH 07/12] lei help: show "NAME=VALUE" properly for -c Eric Wong
@ 2021-03-28 9:01 ` Eric Wong
2021-03-28 9:01 ` [PATCH 09/12] t/lei_store: ensure LeiSearch responds to ->isrch Eric Wong
` (3 subsequent siblings)
11 siblings, 0 replies; 14+ messages in thread
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
This means "lei blob" gets shell completion, too.
---
lib/PublicInbox/LEI.pm | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index bad7fad9..a4f4e58c 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -249,7 +249,12 @@ my %OPTDESC = (
"and\xa0'[]'\x{a0}ranges",
'verbose|v+' => 'be more verbose',
'external!' => 'do not use externals',
-'solve!' => 'do not attempt to reconstruct blobs from emails',
+'mail!' => 'do not look in mail storage for OID',
+'cwd!' => 'do not look in git repo of current working directory',
+'oid-a|A=s' => 'pre-image OID',
+'path-a|a=s' => 'pre-image pathname associated with OID',
+'path-b|b=s' => 'post-image pathname associated with OID',
+'git-dir=s@' => 'additional git repository to scan',
'torsocks=s' => ['VAL|auto|no|yes',
'whether or not to wrap git and curl commands with torsocks'],
'no-torsocks' => 'alias for --torsocks=no',
@@ -786,7 +791,7 @@ sub lei__complete {
if (s/[:=].+\z//) { # req/optional args, e.g output|o=i
} elsif (s/\+\z//) { # verbose|v+
} elsif (s/!\z//) {
- # negation: solve! => no-solve|solve
+ # negation: mail! => no-mail|mail
s/([\w\-]+)/$1|no-$1/g
}
map {
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 09/12] t/lei_store: ensure LeiSearch responds to ->isrch
2021-03-28 9:01 [PATCH 00/12] lei blob and some yak-shaving Eric Wong
` (7 preceding siblings ...)
2021-03-28 9:01 ` [PATCH 08/12] lei blob: flesh out help text Eric Wong
@ 2021-03-28 9:01 ` Eric Wong
2021-03-28 9:01 ` [PATCH 10/12] lei blob: add remote external support Eric Wong
` (2 subsequent siblings)
11 siblings, 0 replies; 14+ messages in thread
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
This is needed for SolverGit, at least, and maybe other
bits we share with PSGI in lei.
---
t/lei_store.t | 1 +
1 file changed, 1 insertion(+)
diff --git a/t/lei_store.t b/t/lei_store.t
index 024ff527..db94f6da 100644
--- a/t/lei_store.t
+++ b/t/lei_store.t
@@ -23,6 +23,7 @@ is($sto->add_eml($eml), undef, 'idempotent');
$sto->done;
{
my $es = $sto->search;
+ ok($es->can('isrch'), ref($es). ' can ->isrch (for SolverGit)');
my $msgs = $es->over->query_xover(0, 1000);
is(scalar(@$msgs), 1, 'one message');
is($msgs->[0]->{blob}, $smsg->{blob}, 'blob matches');
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 10/12] lei blob: add remote external support
2021-03-28 9:01 [PATCH 00/12] lei blob and some yak-shaving Eric Wong
` (8 preceding siblings ...)
2021-03-28 9:01 ` [PATCH 09/12] t/lei_store: ensure LeiSearch responds to ->isrch Eric Wong
@ 2021-03-28 9:01 ` Eric Wong
2021-03-28 9:01 ` [PATCH 11/12] lei: drop coderepo placeholders, submodule TODO Eric Wong
2021-03-28 9:01 ` [PATCH 12/12] treewide: shorten temporary filename Eric Wong
11 siblings, 0 replies; 14+ messages in thread
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
Introduce a new LeiRemote wrapper to provide an internal API
which SolverGit expects. This lets us use HTTP/HTTPS endpoints
to reconstruct blobs off patches as we would with local
endpoints, just more slowly...
---
MANIFEST | 1 +
lib/PublicInbox/LEI.pm | 2 +-
lib/PublicInbox/LeiBlob.pm | 16 +++++--
lib/PublicInbox/LeiRemote.pm | 81 ++++++++++++++++++++++++++++++++++++
t/solver_git.t | 16 ++++++-
5 files changed, 110 insertions(+), 6 deletions(-)
create mode 100644 lib/PublicInbox/LeiRemote.pm
diff --git a/MANIFEST b/MANIFEST
index 9048b900..913ce55c 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -196,6 +196,7 @@ lib/PublicInbox/LeiMirror.pm
lib/PublicInbox/LeiOverview.pm
lib/PublicInbox/LeiP2q.pm
lib/PublicInbox/LeiQuery.pm
+lib/PublicInbox/LeiRemote.pm
lib/PublicInbox/LeiSearch.pm
lib/PublicInbox/LeiStore.pm
lib/PublicInbox/LeiToMail.pm
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index a4f4e58c..a94941a9 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -121,7 +121,7 @@ sub index_opt {
my @c_opt = qw(c=s@ C=s@ quiet|q);
my @lxs_opt = (qw(remote! local! external! include|I=s@ exclude=s@ only=s@
- import-remote! no-torsocks torsocks=s),
+ import-remote! no-torsocks torsocks=s),
PublicInbox::LeiQuery::curl_opt());
# we generate shell completion + help using %CMD and %OPTDESC,
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index f44d8af1..8e610efd 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -6,7 +6,7 @@ package PublicInbox::LeiBlob;
use strict;
use v5.10.1;
use parent qw(PublicInbox::IPC);
-use PublicInbox::Spawn qw(spawn popen_rd);
+use PublicInbox::Spawn qw(spawn popen_rd which);
use PublicInbox::DS;
sub sol_done_wait { # dwaitpid callback
@@ -66,7 +66,10 @@ sub do_solve_blob { # via wq_do
}
open my $log, '+>', \(my $log_buf = '') or die "PerlIO::scalar: $!";
$lei->{log_buf} = \$log_buf;
- my $git = $lei->ale->git;
+ my $git = $lei->{ale}->git;
+ my @rmt = map {
+ PublicInbox::LeiRemote->new($lei, $_)
+ } $self->{lxs}->remotes;
my $solver = bless {
gits => [ map {
PublicInbox::Git->new($lei->rel2abs($_))
@@ -74,7 +77,7 @@ sub do_solve_blob { # via wq_do
user_cb => \&solver_user_cb,
uarg => $self,
# -cur_di, -qsp, -msg => temporary fields for Qspawn callbacks
- inboxes => [ $self->{lxs}->locals ],
+ inboxes => [ $self->{lxs}->locals, @rmt ],
}, 'PublicInbox::SolverGit';
$lei->{env}->{'psgi.errors'} = $lei->{2}; # ugh...
local $PublicInbox::DS::in_loop = 0; # waitpid synchronously
@@ -105,8 +108,15 @@ sub lei_blob {
}
return $lei->fail('no --git-dir to try') unless @$git_dirs;
my $lxs = $lei->lxs_prepare or return;
+ if ($lxs->remotes) {
+ require PublicInbox::LeiRemote;
+ $lei->{curl} //= which('curl') or return
+ $lei->fail('curl needed for', $lxs->remotes);
+ $lei->_lei_store(1)->write_prepare($lei);
+ }
require PublicInbox::SolverGit;
my $self = bless { lxs => $lxs, oid_b => $blob }, __PACKAGE__;
+ $lei->ale;
my ($op_c, $ops) = $lei->workers_start($self, 'lei_solve', 1,
{ '' => [ \&sol_done, $lei ] });
$lei->{sol} = $self;
diff --git a/lib/PublicInbox/LeiRemote.pm b/lib/PublicInbox/LeiRemote.pm
new file mode 100644
index 00000000..399fc936
--- /dev/null
+++ b/lib/PublicInbox/LeiRemote.pm
@@ -0,0 +1,81 @@
+# Copyright (C) 2021 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# Make remote externals HTTP(S) inboxes behave like
+# PublicInbox::Inbox and PublicInbox::Search/ExtSearch.
+# This exists solely for SolverGit. It is a high-latency a
+# synchronous API that is not at all fast.
+package PublicInbox::LeiRemote;
+use v5.10.1;
+use strict;
+use IO::Uncompress::Gunzip;
+use PublicInbox::OnDestroy;
+use PublicInbox::MboxReader;
+use PublicInbox::Spawn qw(popen_rd);
+use PublicInbox::LeiCurl;
+use PublicInbox::ContentHash qw(git_sha);
+
+sub new {
+ my ($cls, $lei, $uri) = @_;
+ bless { uri => $uri, lei => $lei }, $cls;
+}
+
+sub isrch { $_[0] } # SolverGit expcets this
+
+sub _each_mboxrd_eml { # callback for MboxReader->mboxrd
+ my ($eml, $self) = @_;
+ my $lei = $self->{lei};
+ my $xoids = $lei->{ale}->xoids_for($eml, 1);
+ if ($lei->{sto} && !$xoids) { # memoize locally
+ $lei->{sto}->ipc_do('add_eml', $eml);
+ }
+ my $smsg = bless {}, 'PublicInbox::Smsg';
+ $smsg->{blob} = $xoids ? (keys(%$xoids))[0]
+ : git_sha(1, $eml)->hexdigest;
+ $smsg->populate($eml);
+ $smsg->{mid} //= '(none)';
+ push @{$self->{smsg}}, $smsg;
+}
+
+sub mset {
+ my ($self, $qstr, undef) = @_; # $opt ($_[2]) ignored
+ my $lei = $self->{lei};
+ my $curl = PublicInbox::LeiCurl->new($lei, $lei->{curl});
+ push @$curl, '-s', '-d', '';
+ my $uri = $self->{uri}->clone;
+ $uri->query_form(q => $qstr, x => 'm', r => 1); # r=1: relevance
+ my $cmd = $curl->for_uri($self->{lei}, $uri);
+ $self->{lei}->qerr("# $cmd");
+ my $rdr = { 2 => $lei->{2}, pgid => 0 };
+ my ($fh, $pid) = popen_rd($cmd, undef, $rdr);
+ my $reap = PublicInbox::OnDestroy->new($lei->can('sigint_reap'), $pid);
+ $self->{smsg} = [];
+ $fh = IO::Uncompress::Gunzip->new($fh);
+ PublicInbox::MboxReader->mboxrd($fh, \&_each_mboxrd_eml, $self);
+ my $err = waitpid($pid, 0) == $pid ? undef
+ : "BUG: waitpid($cmd): $!";
+ @$reap = (); # cancel OnDestroy
+ my $wait = $self->{lei}->{sto}->ipc_do('done');
+ die $err if $err;
+ $self; # we are the mset (and $ibx, and $self)
+}
+
+sub size { scalar @{$_[0]->{smsg}} } # size of previous results
+
+sub mset_to_smsg {
+ my ($self, $ibx, $mset) = @_; # all 3 are $self
+ wantarray ? ($self->size, @{$self->{smsg}}) : $self->{smsg};
+}
+
+sub base_url { "$_[0]->{uri}" }
+
+sub smsg_eml {
+ my ($self, $smsg) = @_;
+ if (my $bref = $self->{lei}->ale->git->cat_file($smsg->{blob})) {
+ return PublicInbox::Eml->new($bref);
+ }
+ $self->{lei}->err("E: $self->{uri} $smsg->{blob} gone <$smsg->{mid}>");
+ undef;
+}
+
+1;
diff --git a/t/solver_git.t b/t/solver_git.t
index 6d4b93c7..2d803d47 100644
--- a/t/solver_git.t
+++ b/t/solver_git.t
@@ -7,7 +7,7 @@ use PublicInbox::TestCommon;
use Cwd qw(abs_path);
require_git(2.6);
use Digest::SHA qw(sha1_hex);
-use PublicInbox::Spawn qw(popen_rd);
+use PublicInbox::Spawn qw(popen_rd which);
require_mods(qw(DBD::SQLite Search::Xapian Plack::Util));
my $git_dir = xqx([qw(git rev-parse --git-dir)], undef, {2 => \(my $null)});
$? == 0 or plan skip_all => "$0 must be run from a git working tree";
@@ -227,8 +227,20 @@ EOF
my $cmd = [ qw(-httpd -W0), "--stdout=$out", "--stderr=$err" ];
my $td = start_script($cmd, $env, { 3 => $sock });
my ($h, $p) = tcp_host_port($sock);
- local $ENV{PLACK_TEST_EXTERNALSERVER_URI} = "http://$h:$p";
+ my $url = "http://$h:$p";
+ local $ENV{PLACK_TEST_EXTERNALSERVER_URI} = $url;
Plack::Test::ExternalServer::test_psgi(client => $client);
+ skip 'no curl', 1 unless which('curl');
+
+ mkdir "$tmpdir/ext" // xbail "mkdir $!";
+ test_lei({tmpdir => "$tmpdir/ext"}, sub {
+ my $rurl = "$url/$name";
+ lei_ok(qw(blob --no-mail 69df7d5 -I), $rurl);
+ is(sha1_hex("blob ".length($lei_out)."\0".$lei_out),
+ $expect, 'blob contents output');
+ ok(!lei(qw(blob -I), $rurl, $non_existent),
+ 'non-existent blob fails');
+ });
}
}
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 11/12] lei: drop coderepo placeholders, submodule TODO
2021-03-28 9:01 [PATCH 00/12] lei blob and some yak-shaving Eric Wong
` (9 preceding siblings ...)
2021-03-28 9:01 ` [PATCH 10/12] lei blob: add remote external support Eric Wong
@ 2021-03-28 9:01 ` Eric Wong
2021-03-28 9:31 ` Eric Wong
2021-03-28 9:01 ` [PATCH 12/12] treewide: shorten temporary filename Eric Wong
11 siblings, 1 reply; 14+ messages in thread
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
"lei blob" supports --git-dir and -C, and checks if the
current directory has a git directory associated with it.
It will likely support submodules in the future.
I'm inclined to believe declaring coderepos in a command-line
tool is needless clutter and users will rarely want to search
for blobs across different projects when on the command-line.
---
lib/PublicInbox/LEI.pm | 9 ---------
lib/PublicInbox/LeiBlob.pm | 1 +
2 files changed, 1 insertion(+), 9 deletions(-)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index a94941a9..8a07a4c8 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -172,15 +172,6 @@ our %CMD = ( # sorted in order of importance/use:
'remove imported messages from IMAP, Maildirs, and MH',
qw(exact! all jobs:i indexed), @c_opt ],
-# code repos are used for `show' to solve blobs from patch mails
-'add-coderepo' => [ 'DIRNAME', 'add or set priority of a git code repo',
- qw(boost=i), @c_opt ],
-'ls-coderepo' => [ '[FILTER_TERMS...]',
- 'list known code repos', qw(format|f=s z), @c_opt ],
-'forget-coderepo' => [ 'DIRNAME',
- 'stop using repo to solve blobs from patches',
- qw(prune), @c_opt ],
-
'add-watch' => [ 'LOCATION', 'watch for new messages and flag changes',
qw(import! kw|keywords|flags! interval=s recursive|r
exclude=s include=s), @c_opt ],
diff --git a/lib/PublicInbox/LeiBlob.pm b/lib/PublicInbox/LeiBlob.pm
index 8e610efd..91098a90 100644
--- a/lib/PublicInbox/LeiBlob.pm
+++ b/lib/PublicInbox/LeiBlob.pm
@@ -2,6 +2,7 @@
# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
# "lei blob $OID" command
+# TODO: this doesn't scan submodules, but maybe it should
package PublicInbox::LeiBlob;
use strict;
use v5.10.1;
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 12/12] treewide: shorten temporary filename
2021-03-28 9:01 [PATCH 00/12] lei blob and some yak-shaving Eric Wong
` (10 preceding siblings ...)
2021-03-28 9:01 ` [PATCH 11/12] lei: drop coderepo placeholders, submodule TODO Eric Wong
@ 2021-03-28 9:01 ` Eric Wong
11 siblings, 0 replies; 14+ messages in thread
From: Eric Wong @ 2021-03-28 9:01 UTC (permalink / raw)
To: meta
File::Temp only requires four 'X' characters (unlike mkstemp(3),
which requires six). So only so only give it 4 to avoid an
80-column violation and maybe save metadata space on FSes.
---
lib/PublicInbox/LeiOverview.pm | 2 +-
lib/PublicInbox/Msgmap.pm | 2 +-
lib/PublicInbox/SolverGit.pm | 2 +-
lib/PublicInbox/TestCommon.pm | 2 +-
lib/PublicInbox/V2Writable.pm | 6 +++---
lib/PublicInbox/Xapcmd.pm | 9 ++++-----
script/public-inbox-edit | 2 +-
script/public-inbox-init | 2 +-
scripts/ssoma-replay | 2 +-
t/check-www-inbox.perl | 2 +-
t/inbox.t | 2 +-
t/nodatacow.t | 2 +-
12 files changed, 17 insertions(+), 18 deletions(-)
diff --git a/lib/PublicInbox/LeiOverview.pm b/lib/PublicInbox/LeiOverview.pm
index 8e26cba4..68f6c792 100644
--- a/lib/PublicInbox/LeiOverview.pm
+++ b/lib/PublicInbox/LeiOverview.pm
@@ -26,7 +26,7 @@ sub _iso8601 ($) { strftime('%Y-%m-%dT%H:%M:%SZ', gmtime($_[0])) }
# we open this in the parent process before ->wq_io_do handoff
sub ovv_out_lk_init ($) {
my ($self) = @_;
- my $tmp = File::Temp->new("lei-ovv.dst.$$.lock-XXXXXX",
+ my $tmp = File::Temp->new("lei-ovv.dst.$$.lock-XXXX",
TMPDIR => 1, UNLINK => 0);
$self->{"lk_id.$self.$$"} = $self->{lock_path} = $tmp->filename;
}
diff --git a/lib/PublicInbox/Msgmap.pm b/lib/PublicInbox/Msgmap.pm
index 826c4b30..16a9a476 100644
--- a/lib/PublicInbox/Msgmap.pm
+++ b/lib/PublicInbox/Msgmap.pm
@@ -46,7 +46,7 @@ sub new_file {
sub tmp_clone {
my ($self, $dir) = @_;
require File::Temp;
- my $tmp = "mm_tmp-$$-XXXXXX";
+ my $tmp = "mm_tmp-$$-XXXX";
my ($fh, $fn) = File::Temp::tempfile($tmp, EXLOCK => 0, DIR => $dir);
PublicInbox::Spawn::nodatacow_fd(fileno($fh));
$self->{dbh}->sqlite_backup_to_file($fn);
diff --git a/lib/PublicInbox/SolverGit.pm b/lib/PublicInbox/SolverGit.pm
index 1d70975e..92106e75 100644
--- a/lib/PublicInbox/SolverGit.pm
+++ b/lib/PublicInbox/SolverGit.pm
@@ -681,7 +681,7 @@ sub solve ($$$$$) {
$self->{todo} = [ { %$hints, oid_b => $oid_want } ];
$self->{patches} = []; # [ $di, $di, ... ]
$self->{found} = {}; # { abbr => [ ::Git, oid, type, size, $di ] }
- $self->{tmp} = File::Temp->newdir("solver.$oid_want-XXXXXXXX", TMPDIR => 1);
+ $self->{tmp} = File::Temp->newdir("solver.$oid_want-XXXX", TMPDIR => 1);
dbg($self, "solving $oid_want ...");
if (my $async = $env->{'pi-httpd.async'}) {
diff --git a/lib/PublicInbox/TestCommon.pm b/lib/PublicInbox/TestCommon.pm
index d36a63aa..d6e090dd 100644
--- a/lib/PublicInbox/TestCommon.pm
+++ b/lib/PublicInbox/TestCommon.pm
@@ -40,7 +40,7 @@ sub tmpdir (;$) {
unless (defined $base) {
($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!);
}
- my $tmpdir = File::Temp->newdir("pi-$base-$$-XXXXXX", TMPDIR => 1);
+ my $tmpdir = File::Temp->newdir("pi-$base-$$-XXXX", TMPDIR => 1);
($tmpdir->dirname, $tmpdir);
}
diff --git a/lib/PublicInbox/V2Writable.pm b/lib/PublicInbox/V2Writable.pm
index 4130a472..0461257f 100644
--- a/lib/PublicInbox/V2Writable.pm
+++ b/lib/PublicInbox/V2Writable.pm
@@ -660,7 +660,7 @@ sub done {
sub write_alternates ($$$) {
my ($info_dir, $mode, $out) = @_;
- my $fh = File::Temp->new(TEMPLATE => 'alt-XXXXXXXX', DIR => $info_dir);
+ my $fh = File::Temp->new(TEMPLATE => 'alt-XXXX', DIR => $info_dir);
my $tmp = $fh->filename;
print $fh @$out or die "print $tmp: $!\n";
chmod($mode, $fh) or die "fchmod $tmp: $!\n";
@@ -772,11 +772,11 @@ sub import_init {
sub diff ($$$) {
my ($mid, $cur, $new) = @_;
- my $ah = File::Temp->new(TEMPLATE => 'email-cur-XXXXXXXX', TMPDIR => 1);
+ my $ah = File::Temp->new(TEMPLATE => 'email-cur-XXXX', TMPDIR => 1);
print $ah $cur->as_string or die "print: $!";
$ah->flush or die "flush: $!";
PublicInbox::Import::drop_unwanted_headers($new);
- my $bh = File::Temp->new(TEMPLATE => 'email-new-XXXXXXXX', TMPDIR => 1);
+ my $bh = File::Temp->new(TEMPLATE => 'email-new-XXXX', TMPDIR => 1);
print $bh $new->as_string or die "print: $!";
$bh->flush or die "flush: $!";
my $cmd = [ qw(diff -u), $ah->filename, $bh->filename ];
diff --git a/lib/PublicInbox/Xapcmd.pm b/lib/PublicInbox/Xapcmd.pm
index e2d67f6a..9791f02c 100644
--- a/lib/PublicInbox/Xapcmd.pm
+++ b/lib/PublicInbox/Xapcmd.pm
@@ -192,7 +192,7 @@ sub prepare_run {
my $dir = dirname($old);
same_fs_or_die($dir, $old);
my $v = PublicInbox::Search::SCHEMA_VERSION();
- my $wip = File::Temp->newdir("xapian$v-XXXXXXXX", DIR => $dir);
+ my $wip = File::Temp->newdir("xapian$v-XXXX", DIR => $dir);
$tmp->{$old} = $wip;
nodatacow_dir($wip->dirname);
push @queue, [ $old, $wip ];
@@ -220,8 +220,7 @@ sub prepare_run {
$src = [ map { "$old/$_" } @old_shards ];
}
foreach my $dn (0..$max_shard) {
- my $tmpl = "$dn-XXXXXXXX";
- my $wip = File::Temp->newdir($tmpl, DIR => $old);
+ my $wip = File::Temp->newdir("$dn-XXXX", DIR => $old);
same_fs_or_die($old, $wip->dirname);
my $cur = "$old/$dn";
push @queue, [ $src // $cur , $wip ];
@@ -291,7 +290,7 @@ sub cpdb_retryable ($$) {
}
sub progress_pfx ($) {
- my ($wip) = @_; # tempdir v2: ([0-9])+-XXXXXXXX
+ my ($wip) = @_; # tempdir v2: ([0-9])+-XXXX
my @p = split('/', $wip);
# return "xap15/0" for v2, or "xapian15" for v1:
@@ -418,7 +417,7 @@ sub cpdb ($$) { # cb_spawn callback
if ($opt->{compact}) {
my $dir = dirname($new);
same_fs_or_die($dir, $new);
- $ft = File::Temp->newdir("$new.compact-XXXXXX", DIR => $dir);
+ $ft = File::Temp->newdir("$new.compact-XXXX", DIR => $dir);
setup_signals();
$tmp = $ft->dirname;
nodatacow_dir($tmp);
diff --git a/script/public-inbox-edit b/script/public-inbox-edit
index 1c6c4e4a..9498038b 100755
--- a/script/public-inbox-edit
+++ b/script/public-inbox-edit
@@ -133,7 +133,7 @@ $mids
}
my %tmpopt = (
- TEMPLATE => 'public-inbox-edit-XXXXXX',
+ TEMPLATE => 'public-inbox-edit-XXXX',
TMPDIR => 1,
SUFFIX => $opt->{raw} ? '.eml' : '.mbox',
);
diff --git a/script/public-inbox-init b/script/public-inbox-init
index e93cab73..335eb476 100755
--- a/script/public-inbox-init
+++ b/script/public-inbox-init
@@ -79,7 +79,7 @@ PublicInbox::Lock::lock_acquire($lock_obj);
# git-config will operate on this (and rename on success):
require File::Temp;
-my $fh = File::Temp->new(TEMPLATE => 'pi-init-XXXXXXXX', DIR => $dir);
+my $fh = File::Temp->new(TEMPLATE => 'pi-init-XXXX', DIR => $dir);
# Now, we grab another lock to use git-config(1) locking, so it won't
# wait on the lock, unlike some of our internal flock()-based locks.
diff --git a/scripts/ssoma-replay b/scripts/ssoma-replay
index cfb0fbd9..70d0081d 100755
--- a/scripts/ssoma-replay
+++ b/scripts/ssoma-replay
@@ -29,7 +29,7 @@ use strict;
use Email::Simple;
use URI::Escape qw/uri_escape_utf8/;
use File::Temp qw/tempfile/;
-my ($fh, $filename) = tempfile('ssoma-replay-XXXXXXXX', TMPDIR => 1);
+my ($fh, $filename) = tempfile('ssoma-replay-XXXX', TMPDIR => 1);
my $msg = Email::Simple->new(do { local $/; <STDIN> });
select $fh;
diff --git a/t/check-www-inbox.perl b/t/check-www-inbox.perl
index eee8adc2..033b90d1 100644
--- a/t/check-www-inbox.perl
+++ b/t/check-www-inbox.perl
@@ -91,7 +91,7 @@ foreach my $p (1..$nproc) {
}
}
-my ($fh, $tmp) = tempfile('www-check-XXXXXXXX',
+my ($fh, $tmp) = tempfile('www-check-XXXX',
SUFFIX => '.gdbm', UNLINK => 1, TMPDIR => 1);
my $gdbm = tie my %seen, 'GDBM_File', $tmp, &GDBM_WRCREAT, 0600;
defined $gdbm or die "gdbm open failed: $!\n";
diff --git a/t/inbox.t b/t/inbox.t
index b7239e6d..0580cd23 100644
--- a/t/inbox.t
+++ b/t/inbox.t
@@ -13,7 +13,7 @@ is($x->base_url, 'http://example.com/test/', 'added trailing slash');
$x = PublicInbox::Inbox->new({});
is($x->base_url, undef, 'undef base_url allowed');
-my $tmpdir = File::Temp->newdir('pi-inbox-XXXXXX', TMPDIR => 1);
+my $tmpdir = File::Temp->newdir('pi-inbox-XXXX', TMPDIR => 1);
$x->{inboxdir} = $tmpdir->dirname;
is_deeply($x->cloneurl, [], 'no cloneurls');
is($x->description, '($INBOX_DIR/description missing)', 'default description');
diff --git a/t/nodatacow.t b/t/nodatacow.t
index 72860d43..9b67c521 100644
--- a/t/nodatacow.t
+++ b/t/nodatacow.t
@@ -15,7 +15,7 @@ SKIP: {
skip 'BTRFS_TESTDIR not defined', $nr unless defined $dir;
skip 'chattr(1) not installed', $nr unless which('chattr');
my $lsattr = which('lsattr') or skip 'lsattr(1) not installed', $nr;
- my $tmp = File::Temp->newdir('nodatacow-XXXXX', DIR => $dir);
+ my $tmp = File::Temp->newdir('nodatacow-XXXX', DIR => $dir);
my $dn = $tmp->dirname;
my $name = "$dn/pp.f";
^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [PATCH 11/12] lei: drop coderepo placeholders, submodule TODO
2021-03-28 9:01 ` [PATCH 11/12] lei: drop coderepo placeholders, submodule TODO Eric Wong
@ 2021-03-28 9:31 ` Eric Wong
0 siblings, 0 replies; 14+ messages in thread
From: Eric Wong @ 2021-03-28 9:31 UTC (permalink / raw)
To: meta
Eric Wong <e@80x24.org> wrote:
> "lei blob" supports --git-dir and -C, and checks if the
> current directory has a git directory associated with it.
> It will likely support submodules in the future.
>
> I'm inclined to believe declaring coderepos in a command-line
> tool is needless clutter and users will rarely want to search
> for blobs across different projects when on the command-line.
Fwiw, we may we end up supporting read-write JMAP AND expose
blob reconstruction as a vendor-specific JMAP function.
But that's not happening for 1.7...
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2021-03-28 9:31 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-28 9:01 [PATCH 00/12] lei blob and some yak-shaving Eric Wong
2021-03-28 9:01 ` [PATCH 01/12] lei: simplify PktOp callers Eric Wong
2021-03-28 9:01 ` [PATCH 02/12] lei init: split out into separate file Eric Wong
2021-03-28 9:01 ` [PATCH 03/12] lei blob: dclose if already failed Eric Wong
2021-03-28 9:01 ` [PATCH 04/12] lei blob: support --no-mail switch Eric Wong
2021-03-28 9:01 ` [PATCH 05/12] lei blob: fail early if no git dirs Eric Wong
2021-03-28 9:01 ` [PATCH 06/12] lei blob: some extra tests Eric Wong
2021-03-28 9:01 ` [PATCH 07/12] lei help: show "NAME=VALUE" properly for -c Eric Wong
2021-03-28 9:01 ` [PATCH 08/12] lei blob: flesh out help text Eric Wong
2021-03-28 9:01 ` [PATCH 09/12] t/lei_store: ensure LeiSearch responds to ->isrch Eric Wong
2021-03-28 9:01 ` [PATCH 10/12] lei blob: add remote external support Eric Wong
2021-03-28 9:01 ` [PATCH 11/12] lei: drop coderepo placeholders, submodule TODO Eric Wong
2021-03-28 9:31 ` Eric Wong
2021-03-28 9:01 ` [PATCH 12/12] treewide: shorten temporary filename Eric Wong
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).