* [PATCH 0/7] various documentation fixes
@ 2019-05-24 2:57 Eric Wong
2019-05-24 2:57 ` [PATCH 1/7] doc: index: fix miscapitalization of "SQLite" Eric Wong
` (6 more replies)
0 siblings, 7 replies; 8+ messages in thread
From: Eric Wong @ 2019-05-24 2:57 UTC (permalink / raw)
To: meta
The `--compact' switch for xcpdb is really useful.
And fix a few other doc and website things while
we're at it...
Eric Wong (7):
doc: index: fix miscapitalization of "SQLite"
doc: daemon: fix manpage section for nginx
doc: don't barf on missing `git set-file-times'
doc: sync .txt mtime to .pod mtime
doc: generate manpages for some Xapian commands
doc: xcpdb: add switch documentation
doc: add URLs for Xapian manpages
Documentation/.gitignore | 1 +
Documentation/extman.perl | 33 ++++++++++++++++++++
Documentation/include.mk | 42 +++++++++++++++++++++++---
Documentation/public-inbox-compact.pod | 13 +++++++-
Documentation/public-inbox-daemon.pod | 4 +--
Documentation/public-inbox-index.pod | 2 +-
Documentation/public-inbox-xcpdb.pod | 38 +++++++++++++++++++----
script/public-inbox-xcpdb | 2 +-
8 files changed, 119 insertions(+), 16 deletions(-)
create mode 100644 Documentation/extman.perl
--
EW
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH 1/7] doc: index: fix miscapitalization of "SQLite"
2019-05-24 2:57 [PATCH 0/7] various documentation fixes Eric Wong
@ 2019-05-24 2:57 ` Eric Wong
2019-05-24 2:57 ` [PATCH 2/7] doc: daemon: fix manpage section for nginx Eric Wong
` (5 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Eric Wong @ 2019-05-24 2:57 UTC (permalink / raw)
To: meta
Oops :x
---
Documentation/public-inbox-index.pod | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Documentation/public-inbox-index.pod b/Documentation/public-inbox-index.pod
index 2e0ff69..6d2a420 100644
--- a/Documentation/public-inbox-index.pod
+++ b/Documentation/public-inbox-index.pod
@@ -11,7 +11,7 @@ public-inbox-index [OPTIONS] INBOX_DIR
public-inbox-index creates and updates the search, overview and
NNTP article number database used by the read-only public-inbox
HTTP and NNTP interfaces. Currently, this requires
-L<DBD::SQlite> and L<DBI> Perl modules. L<Search::Xapian>
+L<DBD::SQLite> and L<DBI> Perl modules. L<Search::Xapian>
is optional, only to support the PSGI search interface.
Once the initial indices are created by public-inbox-index,
--
EW
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH 2/7] doc: daemon: fix manpage section for nginx
2019-05-24 2:57 [PATCH 0/7] various documentation fixes Eric Wong
2019-05-24 2:57 ` [PATCH 1/7] doc: index: fix miscapitalization of "SQLite" Eric Wong
@ 2019-05-24 2:57 ` Eric Wong
2019-05-24 2:57 ` [PATCH 3/7] doc: don't barf on missing `git set-file-times' Eric Wong
` (4 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Eric Wong @ 2019-05-24 2:57 UTC (permalink / raw)
To: meta
The nginx manpage is in section 8.
---
Documentation/public-inbox-daemon.pod | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/Documentation/public-inbox-daemon.pod b/Documentation/public-inbox-daemon.pod
index a3c97d8..47d15de 100644
--- a/Documentation/public-inbox-daemon.pod
+++ b/Documentation/public-inbox-daemon.pod
@@ -37,7 +37,7 @@ This takes an absolute path to a Unix socket or HOST:PORT
to listen on. For example, to listen to TCP connections on
port 119, use: C<-l 0.0.0.0:119>. This may also point to
a Unix socket (C<-l /path/to/http.sock>) for a reverse proxy
-like L<nginx(1)> to use.
+like L<nginx(8)> to use.
May be specified multiple times to allow listening on multiple
sockets.
@@ -77,7 +77,7 @@ Default: 1
=head1 SIGNALS
-Most of our signal handling behavior is copied from L<nginx(1)>
+Most of our signal handling behavior is copied from L<nginx(8)>
and/or L<starman(1)>; so it is possible to reuse common scripts
for managing them.
--
EW
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH 3/7] doc: don't barf on missing `git set-file-times'
2019-05-24 2:57 [PATCH 0/7] various documentation fixes Eric Wong
2019-05-24 2:57 ` [PATCH 1/7] doc: index: fix miscapitalization of "SQLite" Eric Wong
2019-05-24 2:57 ` [PATCH 2/7] doc: daemon: fix manpage section for nginx Eric Wong
@ 2019-05-24 2:57 ` Eric Wong
2019-05-24 2:57 ` [PATCH 4/7] doc: sync .txt mtime to .pod mtime Eric Wong
` (3 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Eric Wong @ 2019-05-24 2:57 UTC (permalink / raw)
To: meta
It's not critical, but it's nice to have for cache-friendliness
(otherwise I would not have written it :P)
I guess I should follow up on getting it into 'git contrib/':
https://public-inbox.org/git/20100702033709.GA6818@burratino/
---
Documentation/include.mk | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/Documentation/include.mk b/Documentation/include.mk
index 27d6ea6..5c5b473 100644
--- a/Documentation/include.mk
+++ b/Documentation/include.mk
@@ -115,7 +115,9 @@ doc: $(docs)
gz-doc: $(gz_docs)
rsync-doc:
- git set-file-times $(docs) $(txt)
+ # /usr/share/doc/rsync/scripts/git-set-file-times{.gz} on Debian systems
+ # It is also at: https://yhbt.net/git-set-file-times
+ -git set-file-times $(docs) $(txt)
$(MAKE) gz-doc
$(RSYNC) --chmod=Fugo=r -av $(rsync_docs) $(RSYNC_DEST)
clean-doc:
--
EW
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH 4/7] doc: sync .txt mtime to .pod mtime
2019-05-24 2:57 [PATCH 0/7] various documentation fixes Eric Wong
` (2 preceding siblings ...)
2019-05-24 2:57 ` [PATCH 3/7] doc: don't barf on missing `git set-file-times' Eric Wong
@ 2019-05-24 2:57 ` Eric Wong
2019-05-24 2:57 ` [PATCH 5/7] doc: generate manpages for some Xapian commands Eric Wong
` (2 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Eric Wong @ 2019-05-24 2:57 UTC (permalink / raw)
To: meta
Otherwise timestamps for .html files get screwed up, too;
and that hurts caching.
---
Documentation/include.mk | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/Documentation/include.mk b/Documentation/include.mk
index 5c5b473..5b793fb 100644
--- a/Documentation/include.mk
+++ b/Documentation/include.mk
@@ -87,13 +87,13 @@ dtxt += $(mantxt)
all :: $(mantxt)
Documentation/%.txt : Documentation/%.pod
- $(podtext) $< $@+ && mv $@+ $@
+ $(podtext) $< $@+ && touch -r $< $@+ && mv $@+ $@
txt2pre = $(PERL) -I lib ./Documentation/txt2pre <$< >$@+ && \
touch -r $< $@+ && mv $@+ $@
Documentation/standards.txt : Documentation/standards.perl
- $(PERL) $< >$@+ && mv $@+ $@
+ $(PERL) $< >$@+ && touch -r $< $@+ && mv $@+ $@
Documentation/%.html: Documentation/%.txt
$(txt2pre)
--
EW
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH 5/7] doc: generate manpages for some Xapian commands
2019-05-24 2:57 [PATCH 0/7] various documentation fixes Eric Wong
` (3 preceding siblings ...)
2019-05-24 2:57 ` [PATCH 4/7] doc: sync .txt mtime to .pod mtime Eric Wong
@ 2019-05-24 2:57 ` Eric Wong
2019-05-24 2:57 ` [PATCH 6/7] doc: xcpdb: add switch documentation Eric Wong
2019-05-24 2:57 ` [PATCH 7/7] doc: add URLs for Xapian manpages Eric Wong
6 siblings, 0 replies; 8+ messages in thread
From: Eric Wong @ 2019-05-24 2:57 UTC (permalink / raw)
To: meta
They're nowhere to be found on Xapian.org, and links to
external services are either too long (for manpages.debian.org)
or have privacy-invasive tracking JS on them.
---
Documentation/.gitignore | 1 +
Documentation/extman.perl | 33 +++++++++++++++++++++++++++++++++
Documentation/include.mk | 34 ++++++++++++++++++++++++++++++++--
3 files changed, 66 insertions(+), 2 deletions(-)
create mode 100644 Documentation/extman.perl
diff --git a/Documentation/.gitignore b/Documentation/.gitignore
index e78a0d3..4d2ca22 100644
--- a/Documentation/.gitignore
+++ b/Documentation/.gitignore
@@ -1,2 +1,3 @@
/public-inbox-*.txt
/standards.txt
+/.x
diff --git a/Documentation/extman.perl b/Documentation/extman.perl
new file mode 100644
index 0000000..a72caca
--- /dev/null
+++ b/Documentation/extman.perl
@@ -0,0 +1,33 @@
+#!/usr/bin/perl -w
+# Copyright (C) 2019 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+# prints a manpage to stdout
+use strict;
+my $xapmsg = 'See https://xapian.org/ for more information on Xapian';
+my $usage = "$0 /path/to/manpage.SECTION.txt";
+my $manpage = shift or die $usage;
+my $MAN = $ENV{MAN} || 'man';
+my @args;
+$manpage = (split('/', $manpage))[-1];
+$manpage =~ s/\.txt\z//;
+$manpage =~ s/\A\.//; # no leading dot (see Documentation/include.mk)
+$manpage =~ s/\.(\d+.*)\z// and push @args, $1; # section
+push @args, $manpage;
+
+# don't use UTF-8 characters which readers may not have fonts for
+$ENV{LC_ALL} = $ENV{LANG} = 'C';
+$ENV{COLUMNS} = '76'; # same as pod2text default
+$ENV{PAGER} = 'cat';
+my $cmd = join(' ', $MAN, @args);
+system($MAN, @args) and die "$cmd failed: $!\n";
+$manpage =~ /\A(?:copydatabase|xapian-compact)\z/ and
+ print "\n\n", $xapmsg, "\n";
+
+# touch -r $(man -w $section $manpage) output.txt
+if (-f \*STDOUT) {
+ open(my $fh, '-|', $MAN, '-w', @args) or die "$MAN -w broken?: $!\n";
+ chomp(my $path = <$fh>);
+ my @st = stat($path) or die "stat($path) failed: $!\n";
+ # 9 - mtime
+ utime($st[9], $st[9], \*STDOUT) or die "utime(STDOUT) failed: $!\n";
+}
diff --git a/Documentation/include.mk b/Documentation/include.mk
index 5b793fb..b064f29 100644
--- a/Documentation/include.mk
+++ b/Documentation/include.mk
@@ -8,6 +8,11 @@ all::
# Maybe it's not worth it to support non-GNU make, though...
RSYNC = rsync
RSYNC_DEST = public-inbox.org:/srv/public-inbox/
+MAN = man
+
+# same as pod2text
+COLUMNS = 76
+
txt := INSTALL README COPYING TODO HACKING
dtxt := design_notes.txt design_www.txt dc-dlvr-spam-flow.txt hosted.txt
dtxt += marketing.txt
@@ -106,6 +111,26 @@ html: $(docs_html)
gz_docs := $(addsuffix .gz, $(docs) $(docs_html))
rsync_docs := $(gz_docs) $(docs) $(docs_html)
+# external manpages which we host ourselves, since some packages
+# (currently just Xapian) doesn't host manpages themselves.
+xtxt :=
+xtxt += .copydatabase.1
+xtxt += .xapian-compact.1
+xtxt := $(addprefix Documentation/.x/, $(addsuffix .txt, $(xtxt)))
+xdocs := $(xtxt)
+xdocs_html := $(addsuffix .html, $(subst .txt,,$(xtxt)))
+gz_xdocs := $(addsuffix .gz, $(xdocs) $(xdocs_html))
+rsync_xdocs := $(gz_xdocs) $(xdocs_html) $(xdocs)
+xdoc: $(xdocs) $(xdocs_html)
+
+Documentation/.x/%.txt::
+ @-mkdir -p $(@D)
+ $(PERL) -w Documentation/extman.perl $@ >$@+
+ mv $@+ $@
+
+Documentation/.x/%.html: Documentation/.x/%.txt
+ $(txt2pre)
+
doc: $(docs)
%.gz: %
@@ -114,14 +139,19 @@ doc: $(docs)
mv $@+ $@
gz-doc: $(gz_docs)
+
+gz-xdoc: $(gz_xdocs)
+
rsync-doc:
# /usr/share/doc/rsync/scripts/git-set-file-times{.gz} on Debian systems
# It is also at: https://yhbt.net/git-set-file-times
-git set-file-times $(docs) $(txt)
- $(MAKE) gz-doc
- $(RSYNC) --chmod=Fugo=r -av $(rsync_docs) $(RSYNC_DEST)
+ $(MAKE) gz-doc gz-xdoc
+ $(RSYNC) --chmod=Fugo=r -av $(rsync_docs) $(rsync_xdocs) $(RSYNC_DEST)
+
clean-doc:
$(RM) $(man1) $(man5) $(man7) $(gz_docs) $(docs_html) $(mantxt)
+ $(RM) $(gz_xdocs) $(xdocs_html) $(xdocs)
clean :: clean-doc
--
EW
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH 6/7] doc: xcpdb: add switch documentation
2019-05-24 2:57 [PATCH 0/7] various documentation fixes Eric Wong
` (4 preceding siblings ...)
2019-05-24 2:57 ` [PATCH 5/7] doc: generate manpages for some Xapian commands Eric Wong
@ 2019-05-24 2:57 ` Eric Wong
2019-05-24 2:57 ` [PATCH 7/7] doc: add URLs for Xapian manpages Eric Wong
6 siblings, 0 replies; 8+ messages in thread
From: Eric Wong @ 2019-05-24 2:57 UTC (permalink / raw)
To: meta
In particular, the '--compact' switch is really useful since it
works without holding the inbox-wide lock for minutes at a time
on giant inboxes (inboxes where copies can take dozens, if not
hundreds of minutes).
---
Documentation/public-inbox-compact.pod | 10 ++++++++
Documentation/public-inbox-xcpdb.pod | 34 ++++++++++++++++++++++----
script/public-inbox-xcpdb | 2 +-
3 files changed, 40 insertions(+), 6 deletions(-)
diff --git a/Documentation/public-inbox-compact.pod b/Documentation/public-inbox-compact.pod
index 7d37f6f..47d950a 100644
--- a/Documentation/public-inbox-compact.pod
+++ b/Documentation/public-inbox-compact.pod
@@ -16,6 +16,16 @@ writing while it operates.
It enforces the use of the C<--no-renumber> option of
L<xapian-compact(1)>
+=head1 OPTIONS
+
+=over
+
+=item --blocksize / --no-full / --fuller
+
+These options are passed directly to L<xapian-compact(1)>.
+
+=back
+
=head1 ENVIRONMENT
=over 8
diff --git a/Documentation/public-inbox-xcpdb.pod b/Documentation/public-inbox-xcpdb.pod
index 5697dcd..e056b54 100644
--- a/Documentation/public-inbox-xcpdb.pod
+++ b/Documentation/public-inbox-xcpdb.pod
@@ -4,7 +4,7 @@ public-inbox-xcpdb - upgrade Xapian DB formats
=head1 SYNOPSIS
- public-inbox-xcpdb INBOX_DIR
+ public-inbox-xcpdb [OPTIONS] INBOX_DIR
=head1 DESCRIPTION
@@ -14,10 +14,28 @@ upgrading to the latest database format supported by Xapian
recover from Xapian database modifications from
L<public-inbox-watch(1)> or L<public-inbox-mda(1)>.
-This is intended for upgrading the database format used by
-Xapian. It DOES NOT upgrade the schema used by the
-public-inbox PSGI search interface (see
-L<public-inbox-index(1)>).
+=head1 OPTIONS
+
+=over
+
+=item --compact
+
+In addition to performing the copy operation, run L<xapian-compact(1)>
+on each Xapian partition after copying but before finalizing it.
+Compared to the cost of copying a Xapian database, compacting a
+Xapian database takes only around 5% of the time required to copy.
+
+Compared to L<public-inbox-compact(1)>, use of this option is
+preferable for gigantic inboxes where the coarse-grained lock
+currently required for L<public-inbox-compact(1)> can cause
+the compaction to take hours at-a-time.
+
+=item --blocksize / --no-full / --fuller
+
+These options are passed directly to L<xapian-compact(1)> when
+used with C<--compact>.
+
+=back
=head1 ENVIRONMENT
@@ -40,6 +58,12 @@ Default: 10000
=head1 UPGRADING
+This tool is intended for admins upgrading Xapian search databases
+used by public-inbox, NOT users upgrading public-inbox itself.
+
+In particular, it DOES NOT upgrade the schema used by the
+PSGI search interface (see L<public-inbox-index(1)>).
+
=head1 CONTACT
Feedback welcome via plain-text mail to L<mailto:meta@public-inbox.org>
diff --git a/script/public-inbox-xcpdb b/script/public-inbox-xcpdb
index badb95e..ef64e58 100755
--- a/script/public-inbox-xcpdb
+++ b/script/public-inbox-xcpdb
@@ -7,7 +7,7 @@ use PublicInbox::InboxWritable;
use PublicInbox::Xapcmd;
use PublicInbox::Admin;
PublicInbox::Admin::require_or_die('-search');
-my $usage = "Usage: public-inbox-xcpdb INBOX_DIR\n";
+my $usage = "Usage: public-inbox-xcpdb [--compact] INBOX_DIR\n";
my $opt = {};
GetOptions($opt, qw(compact), @PublicInbox::Xapcmd::COMPACT_OPT) or
die "bad command-line args\n$usage";
--
EW
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH 7/7] doc: add URLs for Xapian manpages
2019-05-24 2:57 [PATCH 0/7] various documentation fixes Eric Wong
` (5 preceding siblings ...)
2019-05-24 2:57 ` [PATCH 6/7] doc: xcpdb: add switch documentation Eric Wong
@ 2019-05-24 2:57 ` Eric Wong
6 siblings, 0 replies; 8+ messages in thread
From: Eric Wong @ 2019-05-24 2:57 UTC (permalink / raw)
To: meta
Since we go through the effort of hosting these manpages,
link to them.
---
Documentation/public-inbox-compact.pod | 3 ++-
Documentation/public-inbox-xcpdb.pod | 4 +++-
2 files changed, 5 insertions(+), 2 deletions(-)
diff --git a/Documentation/public-inbox-compact.pod b/Documentation/public-inbox-compact.pod
index 47d950a..20f9d47 100644
--- a/Documentation/public-inbox-compact.pod
+++ b/Documentation/public-inbox-compact.pod
@@ -60,4 +60,5 @@ License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
=head1 SEE ALSO
-L<xapian-compact(1)>, L<public-inbox-index(1)>
+L<xapian-compact(1)|https://public-inbox.org/.xapian-compact.1.html>,
+L<public-inbox-index(1)|https://public-inbox.org/public-inbox-index.html>
diff --git a/Documentation/public-inbox-xcpdb.pod b/Documentation/public-inbox-xcpdb.pod
index e056b54..a04fd37 100644
--- a/Documentation/public-inbox-xcpdb.pod
+++ b/Documentation/public-inbox-xcpdb.pod
@@ -79,4 +79,6 @@ License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
=head1 SEE ALSO
-L<copydatabase(1)>, L<public-inbox-index(1)>
+L<copydatabase(1)|https://public-inbox.org/.copydatabase.1.html>,
+L<xapian-compact(1)|https://public-inbox.org/.xapian-compact.1.html>,
+L<public-inbox-index(1)|https://public-inbox.org/public-inbox-index.html>
--
EW
^ permalink raw reply related [flat|nested] 8+ messages in thread
end of thread, other threads:[~2019-05-24 2:58 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-05-24 2:57 [PATCH 0/7] various documentation fixes Eric Wong
2019-05-24 2:57 ` [PATCH 1/7] doc: index: fix miscapitalization of "SQLite" Eric Wong
2019-05-24 2:57 ` [PATCH 2/7] doc: daemon: fix manpage section for nginx Eric Wong
2019-05-24 2:57 ` [PATCH 3/7] doc: don't barf on missing `git set-file-times' Eric Wong
2019-05-24 2:57 ` [PATCH 4/7] doc: sync .txt mtime to .pod mtime Eric Wong
2019-05-24 2:57 ` [PATCH 5/7] doc: generate manpages for some Xapian commands Eric Wong
2019-05-24 2:57 ` [PATCH 6/7] doc: xcpdb: add switch documentation Eric Wong
2019-05-24 2:57 ` [PATCH 7/7] doc: add URLs for Xapian manpages Eric Wong
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).